Conducting a High-Quality Systematic Review

1. Clinical/research question

The research question is a carefully formulated one and is the critical first step in the process of developing an SR. The rationale, objectives, and scope are described in detail to permit for the eventual progression into the next phases of describing eligibility of patients and populations; the intervention, screening method, or diagnostic test; the selection of the comparators; and the selection of primary and secondary outcomes, including surrogates. This step is summarized in both qualitative and quantitative studies in PICOT (Patient/Population – Intervention/Test – Comparison/Comparator – Outcome – Time) format, sometimes including study design. The selected outcomes reflect those pertinent to clinical practice, and not those reported in studies. The research question serves as the guide for the systematic search strategy, study eligibility, and citation selection.

2. Study selection

Search strategy. The design of a search strategy for the selection of eligible studies requires the assistance of an information specialist/librarian with experience in conducting SRs, using accurate search terms and sources to ensure the transparency and reproducibility of the search. Generally, ≥ 2 databases are searched to ensure all eligible studies are included.³² Search sources outside of medical databases include grey literature (eg, website and policy publications), government sources, and nongovernmental documents.³³ The inclusion of grey literature is routinely recommended by some agencies.³⁴ Inclusivity and completeness are key; thus, avoiding the sole use of English-language sources, including searches of clinical trial registries, and contacting authors for incomplete information is advised.³⁴ The search strategy should ideally be peer reviewed, as this is deemed to improve the quality and comprehensiveness of the search strategy.³⁵ The Peer Review of Electronic Search Strategies (PRESS) is a structured tool that includes guidance and checklists for the completion of the peer review process.³⁵ A detailed search strategy for ≥ 1 medical database is generally included in the publication of an SR to assure reproducibility.

Selection of studies. The selection criteria and the selection process (ie, the assessment of citations/references by reviewers, which should be completed at least by 2 reviewers and independently, as well as the approach to divergences) are determined prior to completion of the search. Criteria used for the selection of citations are generally piloted to ensure that studies of relevance are included. The process of study selection is summarized in a PRISMA diagram (Figure).¹ Notably, separate searches may be needed for quantitative SRs if adverse events or harms for an intervention are infrequent and are not adequately assessed.

• Download figure
• Open in new tab
• Download powerpoint

Figure.

PRISMA flow diagram for new systematic reviews, which includes searches of databases, registers, and other sources.¹ PRISMA: Preferred Reporting Items for Systematic reviews and Meta-Analyses.

The included study designs should consider the availability of similar study designs, for example, the availability of RCTs in SRs focusing on interventions. If large, well-designed RCTs are available, including only RCTs in the SR is a consideration. In the absence of RCTs, or where RCTs have small sample sizes or have not been rigorously conducted, observational studies are incorporated into the SR and analyzed separately from RCTs.

3. Data abstraction

The data abstraction process comprises the abstraction of data and a description of the abstraction process, similar to the selection of citations. This typically outlines the reviewers who will complete the data abstraction and whether the abstraction will be conducted in duplicate (ie, 2 reviewers or more) and independently. Overall, the data abstraction section specifies the characteristics that will be used to satisfy the PICOT criteria, including the conduct of the study, outcome measures, and financial support (eg, industry support).³⁶ For example, study population characteristics should include elements such as sex, age, and comorbidities, among others, to determine similarity to the population of interest. The intervention is described to enable comparability; examples include medication formulation, and route and frequency of administration. Outcome features comprise definitions of outcomes and similarity to the outcomes of interest, including the use of actual outcomes (eg, mortality) or surrogate outcomes (eg, disease-free survival instead of overall survival), statistical measures/units, and a description of the scale (eg, validation) and its administration (eg, self-administered or administered by the research team), if applicable. Software options for citation libraries, mapping of selection criteria, and data abstraction are available (eg, Distiller SR, Covidence^37,38), as are examples of manual data abstraction forms (eg, Cochrane Collaboration³⁶).

Assessment of ROB. An assessment of ROB—also referred to as quality of a study—is an essential component of the data abstraction process. Bias is assessed at the levels of study design, conduct, and analysis, and refers to the confidence in the estimates for outcomes that have been generated.³⁹ Limitations in the study design, conduct, or analysis can lead to systematically inaccurate results.⁴⁰ Bias in studies can be classified into 4 general categories⁴¹: (1) selection bias, the process that leads to groups not being comparable (eg, when allocation is according to prognosis); (2) performance bias, the process of providing dissimilar care (eg, when allocation is not concealed); (3) detection bias, wherein an outcome is influenced by knowledge of the intervention, which tends to be more important for subjective outcomes (eg, pain assessment) than objective outcomes (eg, mortality); and (4) attrition bias, which occurs when participants are lost to follow-up, when there are missing data, or when there are deviations from a study protocol, as remaining participants may not be representative of the population.⁴¹ ROB assessments are intended to detect these biases to ensure that outcomes reflect true estimates, as low-quality studies generally exaggerate treatment effects.⁴²

Numerous tools are available to assess ROB for quantitative and qualitative research. ROB assessments for quantitative research can be checklists (eg, the checklist developed by JBI), scales (eg, Jadad scale for RCTs),⁴³ or domain-based (ie, an assessment at different stages of study, such as the Cochrane Collaboration ROB for RCTs).⁴⁴ Checklists provide a variety of quality measures scored individually, whereas scales also provide a total score by summing features, assuming equal weights for each individual feature (although weights may not be equal) or assigning more emphasis to specific features. The latter has been demonstrated previously to be problematic, as pooling studies according to various scales will lead to different estimates of effect and CIs.^42,45 The assessment of domains provides a descriptive summary of and emphasis on individual components that can lead to bias, such as allocation concealment as a measure of performance bias.

The necessity for critical appraisal in qualitative research has been established; however, tools used to assess ROB are thought to represent a unified approach without differentiating the distinct methodological approaches for qualitative research (such as grounded theory, interpretative phenomenology, or discourse analysis) and methods for data collection (such as interviewing, use of focus groups and observations).^46,47 The available tools include checklists and frameworks for ROB assessment.^46,47 Checklists for qualitative studies are similar to those used in quantitative studies, whereas frameworks assess concepts of (1) transferability (ie, the ability to make connections between data and wider community settings); (2) credibility (ie, the appropriateness of participants’ accounts, as interpreted by the researcher); and (3) reflexivity and transparency (ie, the influence of the researcher on the analysis, rather than grounding the analysis and its transparency).⁴⁶

Instruments used to assess ROB (ie, checklists/scoring systems, scales, domain scores) for quantitative SRs have been evaluated. The methods are similar to each other in that their intent is to determine whether results are plausible, without flaws, and permit for the inclusion of other biases that may be specific to the clinical question, such as variable duration times for outcome assessment.⁴⁸ Each method has advantages and disadvantages. An analysis of scoring systems suggested that an overall score may not necessarily correlate with overall quality of a study, and scales may provide different results for the same study.⁴⁵ The components used in checklists differ somewhat, and although all incorporate consistent features such as masking and allocation concealment, do not require a detailed description compared to using domains.⁴⁹ The use of domains is considered to be a standardized approach to ROB, but interrater agreement and time to completion may be variable and lengthy, respectively.^50,51 The more commonly used methods and published advantages and disadvantages specific to these methods are described in Table 2.^40,50-63

Subjectivity and judgment are required in the assessment of ROB of studies in an SR. As an example, in a study addressing transfusion, blinding may not be considered critical, whereas the blinding of participants for subjective outcomes such as pain would be considered critical.⁴⁸ Further, an overall categorization of ROB (ie, high or low ROB) for each study subsequently needs to be determined depending on the importance of each domain on the outcomes.⁴⁴ Conducting dual, independent reviews will limit additional unnecessary subjectivity.

4. Data synthesis

A metaanalysis is a statistical method to collate outcomes of studies in an SR,^64,65 and is conducted when outcomes—as well as the measurement statistics used for those outcomes—are predominantly the same. Metaanalyses have the potential to (1) improve precision, particularly if there are many small studies that cannot provide convincing evidence of the effect of an intervention in isolation; (2) answer new questions not addressed in individual studies; and (3) address controversies from conflicting results of studies as well as explore differences.⁶⁶ The intent to include a metaanalysis is prespecified in an SR protocol and in the registration of the SR. The rationale for selection of an effect measure (the statistic that compares outcome data) in a meta-analysis⁶⁷ is also generally prespecified. Common effect measures include odds ratios (ORs) or risk ratios for dichotomous/binary outcomes and mean differences or standard mean differences for continuous outcome variables. The risk ratio (relative risk) and OR are relative measures, whereas the risk difference is an absolute measure.⁶⁶ Table 3 defines these measures and describes considerations for selection.

Selection of specific summary statistics also depends on whether values are consistent, have the same mathematical properties, and can be easily understood.⁶⁶ The estimate of the effect measure is generally expressed with the degree of uncertainty, such as a CI or standard error.^65,67 A CI provides a range of probabilities within which the true estimate lies, and is a measure of precision as it reflects the adequacy of the sample size used for the true estimate.⁶⁸ Several software options are available to estimate effect measures.^69,70 An assessment of the effect of overall ROB (or aspects that are considered more significant in the assessment, such as allocation concealment) on the effect measure (ie, sensitivity analysis, which is the primary analysis with the substitution of alternate values according to ROB)⁶⁶ permits for the examination of the robustness of the metaanalysis.⁴¹ Subgroup analysis (ie, dividing participants or studies into subgroups, such as an analysis of male vs female individuals or studies of different geographic locations) may be conducted to assess variability or to explore specific questions.⁶⁶

The synthesis model for a metaanalysis can be a fixed-effect or random-effects model.⁷¹ A fixed-effect synthesis presumes that there is a common treatment effect across all study settings, whereas in a random-effects metaanalysis, treatment effects vary from study to study.⁷¹ In a random-effects model, the differences in observed effect sizes are not only due to random error, similar to a fixed-effects model, but also to variation in true treatment effects (referred to as heterogeneity).⁷¹ The summary effect from a fixed-effect model is an estimate of the assumed common underlying treatment effect; in contrast, for the random-effects model, the summary effect is an estimate of the average of the distribution of treatment effects across various study settings.⁷¹ As between-study heterogeneity is common and may not be identifiable, the random-effects model is the standard (ie, default) model for metaanalyses and is conducted if prespecified in a protocol, despite high heterogeneity.⁷²

Certainty of evidence. In addition to individual study assessment, the overall certainty of evidence according to each outcome needs to be conducted, the most used is the method developed by the Grading of Recommendations Assessment, Development, and Evaluations (GRADE) working group (Table 4).^73,74 Evaluating the reliability and validity for the data for each outcome by determining the methods used to assess them in each individual study is required, as the quality of data for an outcome may differ across studies.⁶⁴ For instance, an outcome may be primary for 1 study and be systematically measured, but it may be a secondary or tertiary outcome in another study and may not be as carefully measured.⁶⁴

GRADE categorizes studies into 2 groups: RCTs and observational studies. The former is assumed to be associated with less ROB but can be downgraded in quality depending on the overall ROB as assessed by 4 features: (1) the directness of evidence (whether the outcome directly answers the health question), (2) precision (the extent of confidence in the estimate of effect to support a decision),⁷⁵ (3) the inconsistency of results (differing estimates of treatment effects across studies), (4) and the presence/absence of publication bias. Publication bias refers to studies not being submitted or published because of the strength and direction (ie, negative) of the trial result.⁷⁶ Studies with statistically significant results are more likely to be published and those with negative results often face delayed publication.⁷⁷ Visual (ie, a funnel plot that assesses whether there is asymmetrical representations of studies, representing publication bias) and statistical tests for asymmetry can be conducted to gauge for publication bias. The certainty in the quality of evidence can also be upgraded for observational studies and nonrandomized studies, such as in cases where there is an evident dose-response relationship (Table 4). The low certainty of evidence assigned by GRADE for nonrandomized studies is due to the fact that causation cannot generally be determined by nonrandomized studies. These nonrandomized studies, however, play a considerable role in identifying associations; can be complementary (eg, provide information in different populations), often providing long-term outcomes of benefit (not available in RCTs) or harm; and they may be more reflective of usual practice. Thus, nonrandomized studies may provide higher quality of evidence than RCTs.⁷⁸ The GRADE approach is also used for qualitative research and categorizes absence of methodological limitations (ie, limitations in design or conduct), adequacy of data (ie, richness and quantity), coherence (ie, clarity and rationale of the fit of data), and relevance (ie, applicability).⁷⁹

Overall, certainty in the evidence for an outcome is presented as high, moderate, low, or very low certainty of evidence of effect, with high certainty suggesting that future research studies are unlikely to change the confidence in the estimate of effect and very low certainty representing an uncertain estimate of effect.⁸⁰ GRADEpro (Evidence Prime), a software for GRADE analyses, generates tables that summarize the description of features used in the categorization of the certainty of evidence (ie, the summary of findings table) as well as estimates of effect based on data from metaanalyses.^9,81 In the absence of an estimate from a metaanalysis, a descriptive assessment of overall certainty of evidence can be used.

A completed SR will also be evaluated for its quality and ROB. Two commonly used tools, A Measurement Tool to Assess Systematic Reviews, version 2 (AMSTAR2)⁸² and Risk of Bias in Systematic Reviews (ROBIS)⁸³ use a domain-based assessment of bias for SRs of RCTs and observational studies, and include such items as protocol registration (AMSTAR2), adequacy of the literature search, eligibility of individual studies, ROB of included individual studies, appropriateness of metaanalytical methods, and consideration of ROB during interpretation. AMSTAR2 also includes conflicts of interest and is thought to be considered easier to use, whereas ROBIS requires more expertise.⁸⁴ Prior to conducting an SR, awareness of the requirements of PRISMA for reporting of SRs and those of AMSTAR2 and ROBIS for the evaluation of an SR are of similar importance.