Abstract
Objective. To assess the current state of reporting of pain outcomes in Cochrane reviews on chronic musculoskeletal painful conditions and to elicit opinions of patients, healthcare practitioners, and methodologists on presenting pain outcomes to patients, clinicians, and policymakers.
Methods. We identified all reviews in the Cochrane Library of chronic musculoskeletal pain conditions from Cochrane review groups (Back, Musculoskeletal, and Pain, Palliative, and Supportive Care) that contained a summary of findings (SoF) table. We extracted data on reported pain domains and instruments and conducted a survey and interviews on considerations for SoF tables (e.g., pain domains, presentation of results).
Results. Fifty-seven SoF tables in 133 Cochrane reviews were eligible. SoF tables reported pain in 56/57, with all presenting results for pain intensity (20 different outcome instruments), pain interference in 8 SoF tables (5 different outcome instruments), and pain frequency in 1 multiple domain instrument. Other domains like pain quality or pain affect were not reported. From the survey and interviews [response rate 80% (36/45)], we derived 4 themes for a future research agenda: pain domains, considerations for assessing truth, discrimination, and feasibility; clinically important thresholds for responder analyses and presenting results; and establishing hierarchies of outcome instruments.
Conclusion. There is a lack of standardization in the domains of pain selected and the manner that pain outcomes are reported in SoF tables, hampering efforts to synthesize evidence. Future research should focus on the themes identified, building partnerships to achieve consensus and develop guidance on best practices for reporting pain outcomes.
The comparative effectiveness research movement and regulatory agencies have challenged Outcome Measures in Rheumatology (OMERACT), an international initiative to standardize outcomes in clinical trials in rheumatology, and similar organizations, to establish truthful, discriminative, and feasible patient-important outcomes for randomized controlled trials and non-randomized studies1,2. Partnerships with organizations working on the development and assessment of patient-important outcomes are necessary and overdue3.
The Cochrane Collaboration is perhaps the largest organization that uses outcomes as the basis to synthesize results of interventional trials to provide high quality, independent evidence to patients, clinicians, and other decision makers. A key part of a Cochrane review is the summary of findings (SoF) table, which presents the results of the major outcomes in the systematic review. Up to 7 major outcomes may be included in the SoF table; they should be those that are deemed most important from a patient/policy perspective and represent both benefit and harm4,5. While the guidance provided in the Cochrane Handbook states that outcomes important to both patients and other decision makers should be included6, there is no explicit process for deciding which outcomes, which outcome instruments, and which effect size metric should be reported in the SoF table. Some Cochrane review groups and other groups such as IMMPACT (Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials) have developed guidance specific to their disease area7,8,9,10,11,12,13,14. Since guidance efforts were developed mostly independently, it is not surprising that there are differences; these include different outcome domains, instruments, methods of analysis and transformations, cut points and thresholds for minimally important and clinically important differences, interpretation of clinical or policy relevance, and methods of presentation.
Patient-reported outcomes (PRO) assessing the effect of a health condition on patients’ lives constitute a major proportion of core sets for clinical trial and systematic reviews of health interventions, especially in the case of chronic, painful conditions. The Cochrane PRO methods group has a chapter in the Cochrane Handbook in which PRO are defined as “reports coming directly from patients about how they feel or function in relation to a health condition and its therapy, without interpretation by healthcare professionals or anyone else”15. A checklist for authors on important considerations when describing and assessing PRO in systematic reviews has been developed. The checklist includes considerations such as rationale for the construct, evidence for reliability and validity of the measurement instruments, and interpretability of the result. The OMERACT Filter 2.0 provides a framework for development of core outcome sets, including both the selection of appropriate domains and instruments16. Similarly, the US Food and Drug Administration (FDA) has published guidance on the development of PRO17.
The PRO domain selected for this evaluation was pain, since 23 Cochrane review groups have pain as a major outcome in their Cochrane reviews of an intervention for a chronic painful condition. It is suspected that the results of pain outcomes are not reported in a consistent manner in SoF tables. Some tables focus on the concept of intensity of pain, others on the interference of pain. Within each of these concepts, different instruments are used, and methods of analysis and presentation vary widely.
We conducted 2 studies with these objectives: (1) to assess the current state of reporting of pain outcomes by sampling SoF tables in Cochrane reviews on chronic musculoskeletal painful conditions, and (2) to elicit opinions of key participants on how best to present pain outcomes to clinicians, patients, and policymakers. The results of this work will contribute to a research platform for OMERACT in partnership with other international initiatives involved in outcome measure methodology in pain.
MATERIALS AND METHODS
Study 1: Assessment of Reporting of Pain Outcomes in Cochrane Reviews
On July 17, 2013, we conducted a search of titles/abstracts/keywords of the Cochrane Library using the key word “pain” to identify all intervention reviews (excluding overviews) in defined chronic musculoskeletal painful conditions from Cochrane review groups (Back; Musculoskeletal; and Pain, Palliative, and Supportive Care) that contained a SoF table. These 3 Cochrane review groups are responsible for the conditions that fall within the World Health Organization definition of a chronic musculoskeletal condition: “inflammatory rheumatic diseases such as rheumatoid arthritis, osteoporosis and other bone diseases, osteoarthritis and related conditions, soft-tissue periarticular disorders, back pain”18. We independently extracted data in duplicate on the pain domains and instruments and used Excel 2010 for data management and analysis.
We extracted composite measures if they contained a pain component. If a SoF table reported more than 1 pain outcome and/or instrument, all were extracted. When a SoF table reported only a standardized mean difference (SMD), this was recorded as “no instrument reported.” If a SMD was reported along with the transformation of the SMD to a specific instrument, then the instrument used for the back transformation was extracted.
Study 2: Survey and Interviews of Key Stakeholders Regarding Presentation of Pain Outcomes in Cochrane SoF Tables
The aim of our survey and interviews was to obtain information from patients, clinicians, and methodologists, on the most important aspects to consider when expressing the pain response of trial participants in chronic musculoskeletal pain intervention studies with respect to Cochrane systematic reviews. Participants were asked: (1) Which of the following domains of pain are important for reporting in a SoF table: pain intensity, pain frequency, pain interference with function, or other domains; (2) What is the best way to present measures of change; and (3) What are the important thresholds/cutoffs for identifying responders in (i) change scales and (ii) achieving predefined absolute “states.”
We used a purposive, expert sampling technique to select survey and interview participants to obtain representation from Cochrane review groups, international initiatives involved in outcome measures methodology, patients with painful musculoskeletal conditions, healthcare practitioners, and methodologists. Prospective participants were sent a link to the survey via E-mail and were asked to participate in an interview to provide more detailed comments. Two members of the project team drafted the survey [a rheumatologist/journal editor/systematic reviewer and senior outcomes researcher (PT), and a managing editor and systematic reviewer (LJM)] that consisted of open-ended text responses. We piloted our survey with 3 invitees, and revised it in response to comments. We then administered our survey using SurveyMonkey™, and invitees were sent 2 reminders to complete the survey. The interviews were conducted by a researcher trained in conducting semistructured interviews. The interview guide followed the sequence of the survey (Supplementary Table 1, available online at jrheum.org). Interviews lasted between 30 and 75 minutes. The majority of interviews (21/24, 88%) were recorded, transcribed verbatim, and then coded. Three were coded from notes taken during the interviews because the audio recording equipment malfunctioned.
The text data were analyzed by 1 researcher (LJM) and checked by a second (PT) for themes, using a directed approach of qualitative content analysis19. This directed approach was used because the survey was developed around existing ideas for themes, as identified by OMERACT executive committee members. We first ascertained the incidence of responses representing the pre-defined ideas and then identified the incidence of newly identified topics.
Ethics approval for survey and interviews was obtained from the University of Split, School of Medicine Ethical Committee, Split, Croatia.
RESULTS
Study 1: Assessment of Reporting of Pain Outcomes in Cochrane Reviews
Our search of the Cochrane Library website identified 57 reviews containing a SoF table that assessed interventions for chronic painful musculoskeletal conditions (see Supplementary Figure 1, available online at jrheum.org). All but 1 SoF table (98%, 56 of 57) reported an outcome of pain (Table 1). Half the SoF tables (41, 53%) reported only the word “pain” in the table’s outcome column, and we assumed the domain of interest was pain intensity based on the scales that were reported. All 56 SoF tables that reported pain presented a measure of pain intensity, in either single or multiple domain instruments; 20 different instruments were reported, with the visual analog scale (VAS) being the most frequent (45%). Pain intensity was measured using a continuous scale for all but 5 outcomes: 4 responder analyses and the outcome “number of people with resting pain” (Table 1). Pain interference was reported in 8 SoF tables (5 different instruments), and pain frequency was reported in 1 multiple domain instrument in a single SoF table. No SoF tables reported other aspects of pain, such as pain quality or pain affect14. In 9 (16%) SoF tables, the instrument for measuring pain intensity was not reported, and of these, 6 reported SMD with no back-transformation using a familiar instrument.
Pain outcome domains and instruments reported in included Cochrane summary of findings (SoF) tables.
Study 2: Survey and Interviews of Key Stakeholders on Expressing Pain Outcomes in Cochrane SoF Tables
Forty-five individuals were invited to participate in a more in-depth discussion via survey and/or telephone interview. Thirty-six completed an interview and/or the survey; 10 completed both an interview and a survey. Therefore, responses were obtained from 36/45 (80%) invited individuals. All 24 interviews were conducted by the same person (LJM). Reasons for nonparticipation of invitees were not obtained, but we assumed they were unavailable. Those involved in either interview or survey included: patients with painful chronic musculoskeletal conditions (n = 3); healthcare practitioners and/or researchers with expertise in outcomes measurement representing the following fields: rheumatology (n = 12), occupational therapy (n = 2), physiotherapy (n = 1), neurology (n = 1), pain management (n = 4), pain psychotherapy (n = 2), and statisticians and methodologists with expertise in outcomes measurement (n = 11). The majority of healthcare practitioners also conduct outcomes research and thus fulfilled more than 1 role.
Each respondent was active in 1 or more of the following initiatives (Supplementary Table 2, available online at jrheum.org): ACTTION/IMMPACT (Analgesic, Anesthetic, and Addiction Clinical Trial Translations/Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials)11,12,13,14, COMET (Core Outcome Measures in Effectiveness Trials)20; COSMIN (COnsensus-based Standards for the selection of health Measurement INstruments)21,22; Cochrane (Editorial Unit; review groups: Back; Musculoskeletal; Neuromuscular Disorders; Pain, Palliative and Supportive Care; and methods groups: Applicability and Recommendations; and Patient Reported Outcomes); OMERACT; or VAPAIN (Validation and Application of a core set of patient-relevant outcome domains to assess the effectiveness of multimodal PAIN therapy)23.
Tables 2 to 5 describe the 4 key themes derived from respondents, along with examples of issues that were raised. We completed the COREQ (COnsolidated criteria for Reporting Qualitative studies) checklist to ensure high-quality reporting of the study (details available from corresponding author).
Theme 1: Which concepts (or “domains”) of chronic pain should be included as “core” in Cochrane summary of findings tables?
Theme 2: Criteria for acceptable clinimetrics/psychometrics for core endpoints for inclusion in Cochrane summary of findings tables.
Theme 3: Which “threshold of meaning” should be presented in the summary of findings table?
Theme 4: Establishing a hierarchy of pain outcome instruments.
There was good agreement in some areas, such as pain intensity being an important domain to measure in a chronic painful musculoskeletal condition. Twenty-four of 36 respondents raised the issue of the importance of incorporating the patient perspective in these discussions. Analysis of other topics resulted in a range of responses, which were occasionally contradictory. For example, most respondents agreed that analysis by “responder” is ideal and when available should be presented. In cases when only mean values are available, a few respondents noted that the presentation of absolute change (e.g., treatment group improvement by 2 points more than a control group on a scale of 0–10) is easily interpretable and is useful for presenting results, while some respondents felt that these mean values should not be presented. Explanations for the latter included: the argument that presenting a mean change is not useful because the distribution response is often bimodal making an “average” change meaningless; or that mean change is not easily interpretable by patients. Others noted the importance of ensuring that treatment groups are similar at baseline in order to interpret absolute change.
DISCUSSION
The results of our 2 projects described above provide compelling evidence of the disparate use of pain outcomes, and underpin the need to establish dialogue between participants in the fields of pain measurement and outcome methodology. Such partnerships can advance development of guidance on best practices for expressing the pain response to an intervention in a way that is most meaningful to decision makers.
The SoF table is the hallmark of current Cochrane reviews, and while it is reassuring that all but 1 of the included reviews of interventions in chronic musculoskeletal pain conditions provided an outcome of pain, the results were not presented in a consistent manner. A variety of scales, cut points, and transformations were reported in the domains of pain intensity, frequency, and interference, making it difficult for the readers of Cochrane reviews to make sense of the evidence across reviews. Different pain conditions were included in the analysis and, as noted by the survey/interview respondents, it was not entirely clear whether chronic painful conditions of different etiologies could be reported similarly. As well, the nature of an intervention may affect the choice of key outcome domains.
The lack of reporting of the outcome instrument used seriously limits the interpretation of results; this was a concern in almost one-third of SoF tables (18/57) that did not report the outcome instrument. IMMPACT has published consensus recommendations for using a numerical rating scale (NRS) to measure pain intensity in chronic pain trials11 and to report the proportion of patients who achieve reductions in pain intensity of ≥ 30% and ≥ 50% (reflecting what are proposed as moderate and substantial clinically important differences, respectively)13. For back pain, 1 consensus paper suggests a 30% reduction in pain as minimally important to patients24. In spite of these recommendations, it is notable that the NRS was not reported in a single SoF table. Responder analyses were reported in 2 SoF tables; the timing of the publication of the recommendations (2005 and 2008) may be a reason for the lack of their use.
Seven global pain domains (intensity, frequency, interference, location, affect, quality, and factors associated with pain) were identified when patients with chronic pain were asked to describe their pain in their own words25. An IMMPACT survey of patients with a variety of chronic pain conditions found that within the concept of pain interference, patients identified 19 aspects of pain interference with daily life (e.g., sleep, social relationships, employment, emotional well-being) as being important14. We found that the majority of SoF tables reported on pain intensity, with few assessing pain interference or frequency or any other of the pain (sub) domains identified by IMMPACT. The majority of survey and interview participants in this study raised the importance of including the patient perspective. They also noted that, although the burden on respondents must be taken into account, a more complex measure of the effect of pain, in addition to intensity, should be considered when reporting evidence for the effectiveness of a treatment for chronic pain. This is a strength of the new ICOAP instrument26 (Intermittent and Constant Osteo-Arthritis Pain), which was developed based on focus groups with patients from 4 countries and used modern psychometric approaches as recommended in the OMERACT Filter 2.016. We look forward to seeing this instrument in the updates of these systematic reviews.
The survey and interviews allowed us to generate themes for future research, based on input from a broad group of people. Participants generally agreed on some topic areas but not on others, such as the preferred method for the presentation of results, highlighting differences of opinion.
With OMERACT’s core principle of actively including patients and others in the consensus process, and through building partnerships with key organizations involved in pain and outcome measurement, there is a strong foundation and opportunity now for achieving consensus and developing guidance.
ONLINE SUPPLEMENT
Supplementary data for this article are available online at jrheum.org.
Acknowledgment
Thank you to those who participated in the survey and interviews.
Footnotes
U. Kaiser received a grant from the German Ministry of Education and Research (BMBF 01GY1326); R. Christensen, grants from the Oak Foundation to the Musculoskeletal Statistics Unit at the Parker Institute; J.A. Singh, grants from the Agency for Health Quality and Research Center for Education and Research on Therapeutics (AHRQ CERTs) U19 HS021110, US National Institute of Arthritis, Musculoskeletal and Skin Diseases (NIAMS) P50 AR060772 and U34 AR062891, National Institute of Aging (NIA) U01 AG018947, National Cancer Institute (NCI) U10 CA149950, the resources and the use of facilities at the VA Medical Center at Birmingham, and research contract CE-1304-6631 from the Patient Centered Outcomes Research Institute (PCORI); C.O. Bingham, in part by grants and awards NIAMS P30-AR053503, a Patient-Centered Outcomes Research Institute (PCORI) Pilot Project Program Award (1IP2-PI000737-01), and a PCORI Methodology Award (SC14-1402-10818); J.A. Singh, research grants from Takeda and Savient and consultant fees from Savient, Takeda, Regeneron and Allergan; member of the American College of Rheumatology’s Guidelines Subcommittee of the Quality of Care Committee, and Veterans Affairs Rheumatology Field Advisory Committee; D.E. Beaton, C.O. Bingham, P. Conaghan, L.S. Simon, J.A. Singh, V. Strand, P. Tugwell, and G.A. Wells are members of the executive of OMERACT, an organization that develops outcome measures in rheumatology and receives arms-length funding from 36 pharmaceutical and clinical research companies, but they do not receive financial remuneration for their activities; P. Jüni has received research grants to the institution from Astra Zeneca, Biotronik, Biosensors International, Eli Lilly, and The Medicines Company, and serves as unpaid member of the steering group of trials funded by Astra Zeneca, Biotronik, Biosensors, St. Jude Medical, and The Medicines Company; PT has received honoraria for quality of life consultancy with BMS, chairs data safety monitoring boards for Chelsea and UCB, has mandated for the FDA, co-chairs an industry board of 6 companies (Abbott, BMS, Merck, Roche, Schering Plough, UCB) for a biologics registry of the Ontario Rheumatology Association [Ontario Biologics Research Initiative (arms-length registry that independently collects information)], received a grant from the Canadian Institutes of Health Research (CIHR) matched by funding from these pharmaceutical companies); and receives an honorarium from the CIHR grant. Statements in this report, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of NIAMS, the National Institutes of Health, or the Patient-Centered Outcomes Research Institute, its Board of Governors, or Methodology Committee.
