| Summary: |
The evidence about the effectiveness, safety and cost-effectiveness of health technologies are key inputs for Health Technology Assessment (HTA) processes. Gold standard sources for these come usually from systematic reviews and meta-analysis (SR&MA) of randomized controlled trials (RCTs). Nevertheless, this evidence base might not be without bias or limitations. Therefore, to ensure findings from SR&MA of RCTs can be translated into meaningful recommendations for health decision-makers, effect estimates of health technologies should always be integrated with ratings of the quality of evidence. One of the most widely used frameworks to assess the quality of evidence in SR&MA is the Grading of Recommendations, Assessment, Development, and Evaluations (GRADE). A GRADE judgment about the quality of a body of evidence reflects the confidence that systematic reviewers have on effect estimates considering five domains: risk of bias (RoB) of included studies, inconsistency among studies' results, indirectness in addressing the research question, imprecision in effect estimates, and considerations about publication bias.
Within the GRADE framework, standard methods exist to assess within-study RoB, inconsistency, imprecision, and publication bias. However, a systematic and transparent approach for evaluating the domain of indirectness is notably absent. The lack of systematic tools, methods and literature addressing this gap might undermine the validity and reliability of GRADE judgments on indirectness, consequently affecting the trustworthiness of overall GRADE ratings in SR&MA. Considering the crucial role that overall GRADE ratings play in influencing regulatory and HTA decisions (e.g. policymakers often prioritize interventions and allocate resources based on the quality of evidence), it becomes imperative to establish systematic and transparent processes for GRADE judgments on indirectness, aligning them with those used in the other domains. The research p  |
Summary
The evidence about the effectiveness, safety and cost-effectiveness of health technologies are key inputs for Health Technology Assessment (HTA) processes. Gold standard sources for these come usually from systematic reviews and meta-analysis (SR&MA) of randomized controlled trials (RCTs). Nevertheless, this evidence base might not be without bias or limitations. Therefore, to ensure findings from SR&MA of RCTs can be translated into meaningful recommendations for health decision-makers, effect estimates of health technologies should always be integrated with ratings of the quality of evidence. One of the most widely used frameworks to assess the quality of evidence in SR&MA is the Grading of Recommendations, Assessment, Development, and Evaluations (GRADE). A GRADE judgment about the quality of a body of evidence reflects the confidence that systematic reviewers have on effect estimates considering five domains: risk of bias (RoB) of included studies, inconsistency among studies' results, indirectness in addressing the research question, imprecision in effect estimates, and considerations about publication bias.
Within the GRADE framework, standard methods exist to assess within-study RoB, inconsistency, imprecision, and publication bias. However, a systematic and transparent approach for evaluating the domain of indirectness is notably absent. The lack of systematic tools, methods and literature addressing this gap might undermine the validity and reliability of GRADE judgments on indirectness, consequently affecting the trustworthiness of overall GRADE ratings in SR&MA. Considering the crucial role that overall GRADE ratings play in influencing regulatory and HTA decisions (e.g. policymakers often prioritize interventions and allocate resources based on the quality of evidence), it becomes imperative to establish systematic and transparent processes for GRADE judgments on indirectness, aligning them with those used in the other domains. The research plan we propose seeks to make a meaningful contribution by addressing this imperative.
Based on the approach used for GRADE ratings on within-study bias, which involves assessments at the study level, we advocate that a similar methodology should be applied to rate the indirectness domain. With this end in view and recognizing the relevance of RCTs as gold standards for SR&MA, our research plan aims to develop and validate an instrument for assessing, at the study-level, the extent to which RCTs are subject to indirectness - the INDIRECT tool. To achieve this goal, we will start by creating a comprehensive list of items for potential inclusion in the INDIRECT tool. A methodological review will be conducted to identify studies exploring, describing, or analyzing items related to indirectness in RCTs. Additionally, Summary of Findings tables included in Cochrane Systematic Reviews will
be screened to identify further items flagged by systematic reviewers as reasons for downgrading the quality of evidence due to indirectness within the context of the GRADE framework. A final list of items will be presented at an expert panel meeting where panelists will select the most appropriate set of items to be included in the tool and reach consensus on its final structure. The next stage of the research plan will comprise analysis aimed at evaluating the measurement properties of the INDIRECT tool. Initially, an inter-rater reliability study will be conducted, during which independently raters will apply the INDIRECT tool to a set of 400 to 600 RCTs sourced from Cochrane Systematic Reviews. Subsequently, item response theory models will be fitted to the response pattern data obtained in the reliability study, with the aim of assessing the validity of the tool and generating continuous indirectness scores for each assessed RCT. Finally, 3-class Bayesian latent class models and three-way Bayesian receiver
operating characteristic (ROC) surfaces will be used to establish cutoffs for the indirectness score.
The cutoffs will enable discrimination between RCTs with no issues regarding indirectness, serious indirectness issues, or very serious indirectness issues. Finally, to enhance the applicability of
the INDIRECT tool we will deploy it as an open-source web-application. In conclusion, upon successful completion of this project, we expect to provide a robust instrument - the INDIRECT tool - delivered in a web-application platform to measure indirectness in RCTs. Our major goal is to enhance the accuracy, consistency, and transparency of GRADE ratings, ultimately improving HTA processes globally. |