“Evidence-based medicine” (EBM) stresses examining evidence from clinical research1 as the preferred method of clinical decision making, de-emphasizing intuition, unsystematic clinical experience, and pathophysiologic rationale. Likewise, evidence-based practice (EBP) holds that evidence should be the basis for particular interventions and management plans that are likely applicable to most patients. Logically, this practice would demand that experts develop evidence-based recommendations founded on valid, reliable, and transparent systematic reviews and/or metaanalyses2,3.
In this issue of The Journal, Roubille, et al present recommendations for the management of comorbidities, focusing on 8 areas within rheumatoid arthritis (RA), psoriasis (PsO), and psoriatic arthritis (PsA), based on a review of 407 articles4. Their report summarizes the results of the Canadian Dermatology-Rheumatology (DR) Comorbidity Initiative’s systematic literature searches and consensus-based recommendations from a meeting held in Toronto in 2013, sponsored by the pharmaceutical company AbbVie4. The authors report that they did a thorough, systematic review, followed by data extraction and subsequent metaanalyses (including forest plots summarizing the adjusted relative risk estimates, etc.). However, the authors apparently do not want to reveal their explicit findings (yet!), stating that the details and results of the systematic literature review for each topic will be published separately4. We have no reason to question the integrity or the content of the recommendations that came out of this work, but we worry about a possible trend that could encourage guideline panels, etc., to publish their evidence-synthesis secondarily to their recommendation while claiming in a peer-reviewed journal (like The Journal of Rheumatology) that their work represents evidence-based recommendations.
Although realizing that we might be perceived as having intellectual conflicts of interest (i.e., being editors in the Cochrane Collaboration), we would like to encourage systematic and explicit methods of making judgments because they will likely reduce errors and improve communication5. The aim of the Cochrane Collaboration is to help healthcare providers, patients, patient advocates and carers, and policy makers arrive at well-informed decisions on healthcare treatments by preparing, maintaining, and disseminating methodologically strong systematic reviews (http://www.cochrane.org/). Briefly, the Cochrane Collaboration’s mission is to provide accessible, credible information to support informed decision-making. We strongly believe that users of clinical practice guidelines and other recommendations should know how much confidence they can place in recommendations. Roubille, et al4 did not report the actual data; instead — “saving the evidence” for later, subsequent papers. We find, therefore, that their claim in a peer-reviewed journal that their recommendations are evidence-based is not supported, as their recommendations are missing any presentation of the actual evidence to the peer reviewer or to users of these guidelines.
We would ask, rhetorically: Do evidence-based recommendations need to reveal their evidence? According to the current thinking in EBM, the answer is probably ‘yes.’ The Grades of Recommendation, Assessment, Development, and Evaluation (GRADE) Working Group has developed a system of rating quality of evidence that improves reliability in comparison to intuitive judgments about the evidence6. One of the key principles behind the GRADE methodology is the aim to make it easier for users to assess the judgments behind recommendations7,8.
Evidence summaries (e.g., based on forest plots, etc.) should be available to the user of a recommendation, enabling a fair judgment of the quality of evidence supporting the strength of recommendations. Ideally, the summary would consist of full GRADE evidence profiles or a summary-of-findings (SoF) table based on a systematic review9. At a minimum, the evidence that was assessed and the methods that were used to identify and appraise it should be clearly described. Making recommendations matter to end users requires specifying a priori the relevant setting, population, intervention, comparator, and the relative importance of the outcomes before gathering the evidence and again when evidence summaries are complete10.
Judging the Quality of the Evidence
Evidence from randomized trials without important limitations constitutes high-quality evidence, whereas observational studies without special strengths or important limitations constitute low-quality evidence. However, while looking at all the evidence after an appropriate “meta-strategy,”4 evidence-based recommendations should explicitly delineate each of the following criteria that can lower the quality of the evidence (i.e., a consequence of reduced confidence in the estimate of effect)11:
Risk of bias/study limitations12
Risk of publication bias13
Imprecision14
Inconsistency of results15
Indirectness16.
Any of these factors could mandate that we rate down the quality of the evidence. On the other hand, if observational studies are also included in the evidence-synthesis, extra criteria (e.g., a large magnitude of effect) might apply when rating the overall quality of the evidence17, as their presence might increase our confidence in the estimate of effect18.
We believe that end users of evidence-based recommendations require succinct, transparent, easily digested evidence summaries. The GRADE process facilitates the creation of summaries in the form of evidence profiles and/or SoF tables5. It is important to keep in mind that the judgment available in an SoF table (usually) does not refer to individual studies but rather to a body of evidence. An SoF table presents the credibility of all the available evidence answering a specific clinical question; that is, the body of evidence — such as from a number of trials with good internal validity — may be associated with a low risk of bias, but our confidence in the estimate may be compromised by a number of other factors (as illustrated above)11.
Although the quality of evidence that goes into an evidence-based recommendation spans a continuum, the approach recommended by GRADE results in either high, moderate, low, or very low confidence in the estimate for any given clinical question6.
GRADE provides definitions on how to interpret the different levels of evidence11, which have implications for researchers and funding bodies that are considering whether a research question is valuable or not:
High quality evidence. We are very confident that the true effect lies close to the estimated effect (i.e., no further studies are needed).
Moderate quality evidence. We feel confident in the estimated effect, but it might be different from the true effect (i.e., we still have some research we would like to see).
Low quality evidence. Our confidence in the estimated effect is limited, as the true effect may be substantially different (i.e., we need rigorous prospective studies).
Very low quality evidence. We have very little confidence in the proposed estimated effect, and the true effect is likely substantially different (i.e., at present the evidence is mostly based on opinions and/or unsystematic clinical observations).
Determining the Strength of a Recommendation
We strongly recommend that all organizations adhere to the GRADE process (or a similar transparent approach) when developing recommendations regarding estimation of benefit and harm, as well as when judging the quality of evidence. With the GRADE approach, the strength of a recommendation reflects the extent to which we can be confident that the desirable effects of a management strategy outweigh the undesirable effects. For example, the European League Against Rheumatism (EULAR) states in its standardized operating procedures for the elaboration, evaluation, dissemination, and implementation of recommendations (updated recently)19 that the GRADE system should be used for guidance throughout the elaboration of clinical recommendations. The American College of Rheumatology (ACR) makes a similar recommendation. Unfortunately, EULAR still endorses a system that is difficult to remember and communicate when writing recommendations19. Unlike other previous grading systems, the GRADE approach has only 2 categories for recommendations: either strong or conditional (either for or against). A strong recommendation reflects our confidence that the desirable effects of adherence outweigh the undesirable effects20. Conditional recommendations, on the other hand, indicate that although the desirable effects of adherence probably outweigh the undesirable effects, the panel is less confident that health care professionals will recognize that different choices will be appropriate for different patients. Conditional recommendations for a given intervention imply the importance of shared decision making for delivering patient-centered care; healthcare professionals must help each patient to arrive at a management decision consistent with the patient’s values and preferences.
In this issue of The Journal, Roubille, et al present thorough work, based on expert opinions of the Canadian Dermatology-Rheumatology Comorbidity Initiative4. As the authors do not report the details and results of their systematic reviews, we do not support the claim that this report represents evidence-based recommendations at this stage20. Perhaps once the evidence synthesis is published and made available to the users, it might meet this requirement and earn this label.
Footnotes
The Musculoskeletal Statistics Unit at the Parker Institute (RC) is supported by grants from the Oak Foundation. JAS is supported by grants from the Agency for Health Quality and Research Center for Education and Research on Therapeutics (U19 HS021110), US National Institute of Arthritis, Musculoskeletal and Skin Diseases (P50 AR060772 and U34 AR062891), National Institute of Aging (U01 AG018947), National Cancer Institute (U10 CA149950), and the Patient Centered Outcomes Research Institute (research contract CE-1304-6631).
