Abstract
Objective. The aim of this study was to develop a patient-centered quality measurement framework to address a predefined vision statement and 7 strategic objectives for rheumatoid arthritis (RA) care that was developed in prior qualitative work with arthritis stakeholders.
Methods. One hundred forty-seven RA-related performance measures (PMs) were identified from a systematic review. A candidate list of 26 PMs meeting predefined criteria and addressing the strategic objectives previously defined was then assessed during a 3-round (R) modified Delphi. Seventeen panelists with expertise in RA, quality measurement, and/or lived experience with RA rated each PM on a 1–9 scale based on the items of importance, feasibility, and priority for inclusion in the framework during R1 and R3, with a moderated discussion in R2. PMs with median scores ≥ 7 on all 3 items without disagreement were included in the final set, which then underwent public comment.
Results. Twenty-one measures were included in the final framework (15 PMs from the Delphi and 6 published system-level measures on access to care and treatment). The measures included 4 addressing early access to care and timely diagnosis, 12 evidence-based care for RA and related comorbidities, 1 addressing patient participation as an informed partner in care, and 4 on patient outcomes.
Conclusion. The proposed framework builds upon existing measures capturing early access to care and treatment in RA and adds important PMs to promote high-quality RA care and outcome measurement. In the next phase, the authors will test the framework in clinical practice in addition to addressing certain areas where no suitable PMs were identified.
Rheumatoid arthritis (RA) affects approximately 1% of the population1,2 and is the most prevalent autoimmune inflammatory arthritis. Early and targeted treatment strategies are key to improved patient outcomes and are central principles described in current RA guidelines3,4,5,6. RA is also associated with other consequences of systemic inflammation including an increased risk of infection7, cardiovascular (CV) disease7,8,9, and osteoporosis10. Addressing disease activity and the additional consequences associated with the disease is central to optimizing patient outcomes.
To support efforts to provide high-quality, comprehensive RA care, a “Pan-Canadian Approach to Inflammatory Arthritis Models of Care” was developed by the Arthritis Alliance of Canada (AAC) in 201411. The model describes 6 key elements including identification, access, medical management, shared care, patient self-management, and patient and system performance measurement to inform quality improvement11. In 2016, in collaboration with the AAC, the first System-level Performance Measures (PMs) for Inflammatory Arthritis care in Canada were developed to support quality improvement, research efforts, and advocacy12.
The set included 6 measures addressing early access to care and treatment and have been subsequently tested in different data sources across 5 Canadian provinces13,14,15,16 and a national early inflammatory arthritis cohort17. The measures address wait times for rheumatologist care and percentage of patients seen within a year of diagnosis, time to disease-modifying antirheumatic drugs (DMARD), percentage of patients treated with a DMARD, percentage of patients seen on a yearly basis, and rheumatologists per capita12. The measures reflect important aspects of care that have been shown to improve outcomes; however, patient outcomes were not addressed by the initial measurement set, nor were processes of care beyond DMARD use and yearly follow-up.
Given a growing interest in measuring quality of care and patient outcomes in a standardized fashion, in collaboration with the AAC and the Canadian Rheumatology Association (CRA), we conducted focus groups and semistructured interviews with rheumatologists, allied healthcare providers, researchers, people living with RA, clinic managers, and healthcare leaders to determine their perspectives on RA quality of care18. This work defined a vision statement (“Ensuring patient-centered, high-quality care for people living with RA”). We further developed 7 strategic objectives for a national quality measurement framework for RA18 derived from the following themes: (1) early access and timeliness of care; (2) high-quality care for the ongoing management of RA and comorbidities; (3) patient self-management tools and educational materials for shared decision making; (4) multidisciplinary care; (5) patient outcomes; and (6) patient experience and satisfaction with care. In the present study we describe the selection of PMs for inclusion in the framework to address the strategic objectives.
MATERIALS AND METHODS
Selection of candidate PMs. An overview of the study’s methodology is presented in Figure 1. The primary source of candidate PMs considered for the framework was a previously published systematic review of quality measures for inflammatory arthritis19. While this review identified PMs for several inflammatory arthritis conditions, only those specific to RA (n = 143) or addressing inflammatory arthritis in general (n = 4) were considered for the current study. To ensure no recently published PMs were omitted from consideration, an update of the original search19 was conducted on July 11, 2018. The articles were independently reviewed by 2 reviewers to decide whether the RA quality measures were described in sufficient detail to be considered for inclusion in the framework.
Overview of methods for framework development and performance measure (PM) selection.
A classification exercise was conducted by 2 team members (CEB and VB), whereby the full set of RA PMs identified from the systematic review were mapped to the 6 strategic objectives defined in Phase 1. Each PM on the list was also classified according to which element of the original AAC model of care they addressed11; whether the measure addressed the structure, processes, or outcome of care, according to the Donabedian classification for measuring quality of care20; and last, which of the Institute of Medicine Domains (IOM) of Quality it addressed (e.g., safety, effectiveness, patient-centeredness, timeliness, efficiency, and equity)21. Four team members (CEB, VB, KT, and MH) then excluded PMs that did not align with the vision and strategic objectives of the overall framework; reasons for exclusion were recorded.
Following candidate PM selection, a working document was prepared outlining each measure (including proposed numerator, denominator, and exclusions) and the results of the classification exercise (e.g., AAC, IOM, and Donabedian). The current guidelines for the CRA3,4, American College of Rheumatology5, European League Against Rheumatism6,22,23, and National Institute for Health and Clinical Care Excellence guidelines24 were assessed for any gaps in measurement and reviewed for supporting recommendations for each PM. Targeted searches were then conducted to identify candidate measures used in non-RA populations that could be relevant to RA care, where no existing PMs were identified to measure concepts identified as important in the proposed framework.
Modified Delphi consensus process to select PMs. A panel consisting of 15 RA and measurement experts and 2 people living with RA was established. A purposive sample of measurement experts (based on a review of publications on quality measurement, quality improvement, and/or guideline development in the last 10 years in Canada) were recruited by email and selected to ensure regional diversity. People living with arthritis who have been part of previous measure development exercises or quality improvement efforts were recruited through arthritis consumer organizations.
A 3-round modified Delphi consensus process to select PMs for inclusion was carried out over 2 months (September and October 2018)25 using a Qualtrics electronic survey. In Round 1, panelists were asked to rate each candidate PM on the following criteria: (1) Does the measure target an important gap in RA care (1 = gap not at all important; 9 = gap very important); (2) How likely is it that the information required to report this measure will be available in a typical Canadian health system (1 = highly unlikely; 9 = highly likely); and (3) Overall priority of including this item in the framework (1 = unnecessary; 9 = essential). A single free-text field was provided at the end of the survey in Round 2 and under each PM in Round 3 for entering additional narrative comments or questions.
In Round 2, participants were invited to review their responses in comparison to the aggregated responses for each PM and to discuss results and discrepancies with the group. Round 2 was conducted by teleconference and was moderated by a clinician-scientist with extensive experience in quality measurement as well as with Delphi methodology (CEB).
In Round 3, participants re-rated all measures electronically using the same criteria and response choices as described above for Round 1 while considering their previous ratings, group ratings, and Round 2 group discussions.
Inclusion in the framework required median scores ≥ 7 on all 3 items (important care gap, data availability, overall priority for inclusion) with no disagreement among participants. Disagreement was calculated according to the RAND/UCLA Appropriateness Method handbook26. Disagreement was defined according to the interpercentile range adjusted for symmetry (IPRAS), which is calculated using the following formula26: IPRAS = 2.35 + [Asymmetry Index (AI)*1.5].
Public comment on final included indicators. Following the modified Delphi process to select the PMs, we solicited public comments in collaboration with the CRA and the AAC. For approximately 1 month, documents were posted on a public-access section of the CRA Website. Material presented for review included a description of the methods for Phase 1 (focus groups and interviews) and Phase 2 (modified Delphi consensus process), and a draft version of the RA quality framework (PM, vision, and strategic objectives). Notification of the opportunity to review and provide comment on these materials included an initial and reminder “e-blast” (messages sent through email by the CRA to its members including rheumatologists, researchers, trainees, and emeritus members), and a notice in the AAC newsletter sent to members, which included a link to the material on the CRA website. In addition, all participants in the Phase 1 focus groups and interviews were invited to participate in the public comment.
Feedback was anonymous using a Web form designed and administered by the CRA. No identifying information was requested, and comments were forwarded to the research team once received by the CRA through the Web form. Comments were considered by members of the measure development team for any final changes or clarifications to wording of the PM or the framework.
Ethics. This project was approved by the Conjoint Health Research Ethics Board at the University of Calgary (REB16-0556). All participants in the modified Delphi consensus process provided their consent to be involved in this project.
RESULTS
Selection of candidate performance measures. The systematic review to identify candidate PMs was updated and 6 articles were identified for full-text review; however, none were suitable for inclusion in the framework. Thus, the 147 RA-related PMs included in the original systematic review were evaluated for inclusion. Of those, 6 were AAC system-level PMs12, which were deemed essential to the framework and were not subjected to the Delphi process. Through the evaluation of the remaining 141 candidate PMs, 119 were removed and 22 were retained (reasons for exclusion shown in Figure 2). Through the categorization exercise, some of the strategic objectives of the framework were not completely addressed by existing RA PMs, including vaccination, pain assessment, and access to multidisciplinary care, leading to the addition of 4 measures in these areas identified from targeted reviews and adapting the measures for RA. The final list presented to the Delphi panelists consisted of 26 PMs (complete list shown in the Supplementary Methods, available with the online version of this article).
Flow diagram of selection of performance measures (PMs) for inclusion in the framework. AAC: Arthritis Alliance of Canada; ACR: American College of Rheumatology; RA: rheumatoid arthritis.
Modified Delphi consensus process to select framework PMs. The Delphi panel consisted of 10 (59%) rheumatologists, 4 (23%) physiotherapists, 2 (12%) people living with RA, and 1 (6%) health services researcher. Of note, some individuals had expertise based on more than 1 role (e.g., researchers who were also healthcare providers or had roles in quality improvement). The group represented 6 Canadian provinces. All 17 (100%) of the invited panelists participated in Round 1 and Round 3 of the online modified Delphi process, which were conducted online. Nine (53%) panelists participated in the Round 2 teleconference (2 rheumatologists, 4 physiotherapists, 2 people living with RA, 1 health service researcher), but all had opportunity to provide feedback by email and received a summary of the discussion from Round 2.
Following Round 2, ten PMs were modified based on the feedback from the panelists, and no new PMs were added as candidates for consideration. Details of the modifications and rationale are presented in Supplementary Table 1 (available with the online version of this article). A majority of modifications were minor wording suggestions to ensure measures were applicable to the Canadian context to most rheumatology practices. For example, measures addressing disease activity referring only to the Disease Activity Score in 28 joints (DAS28) were adjusted to ensure that other appropriate and validated disease activity scores widely used in Canada could be applied instead (e.g., the Clinical Disease Activity Index [CDAI] or Simplified Disease Activity Index).
In Round 3, of the 26 measures, 15 (58%) reviewed during the Delphi process met inclusion criteria for the final set (Table 1). Therefore, along with the existing 6 AAC measures, the proposed framework included 21 measures (Figure 2). In the final framework, the wording of individual PM was harmonized for ease of communication. In addition, in Table 2, there is an outline of the “fit” of each PM with each strategic objective and alignment with the domains of quality defined by the Canadian Institutes of Health Information27, a derivation of the IOM quality framework21 used nationally.
Final voting results on 26 candidate PMs.
RA quality measurement framework showing PMsa categorized according to strategic objective and mapped to the CIHI quality domain.
Importantly, in the final set of measures, none were selected through the Delphi process to address strategic objective 4 (“to provide access to multidisciplinary healthcare providers with training and expertise in the assessment and management of RA”) or strategic objective 6 (“to measure and optimize patient experience & satisfaction with care”). This was due to panelist concerns around the feasibility of measurement of the PMs proposed to address these objectives.
Phase 4: Public comment. During the month-long public comment process, 18 individuals anonymously submitted comments. Some of the comments addressed opinions about measure inclusion. For example, 1 individual was concerned about the inclusion of measures addressing BMI, lifestyle counseling, and CV screening, as they did not feel these were within the purview of the rheumatologist and were better addressed in primary care. In contrast, another individual felt there were not enough indicators addressing RA comorbidities.
Other concerns were raised around the wording of some of the measures including the terms “treatment plan” (% of RA patients with a treatment plan developed between them and their clinician/health professionals at each visit), “rheumatology team” (% of RA patients under the care of a rheumatology team seen in follow-up by a rheumatology team member at least once per year), and “follow-up plan” (% of visits for RA patients with documentation of a pain assessment using a standardized tool[s] on each visit AND documentation of a follow-up plan when pain is present). The present wording has been retained in the final measures to allow for further operationalization depending on the model of care (e.g., nurse-led vs rheumatologist-led) and clinic-specific practices for documentation. As highlighted by the public comment process, there are no existing measures to precisely identify if “treatment plan” or “follow-up plan” involved shared decision making, highlighting an area in need of additional study to better address strategic objective 3 (“to provide patients with the right information at the right time to be able to participate as informed partners in their care and be supported to self-manage as appropriate”).
While some individuals felt the framework was too long, others noted additional “missing” measures including a herpes zoster vaccine measure, osteoporosis treatment measures (to complement the existing screening measure), a cost-effectiveness measure for biologics, and a measure addressing access to allied healthcare providers. These important gaps are noted and can be reassessed in further iterations of the framework.
A final concern was the feasibility of disease activity measurement outside of a research context. For example, some measures encouraged using DAS28 scoring, while CDAI may be more feasible in many centers. All affected measures were adjusted in their specifications to allow for CDAI documentation of disease activity. Limitations expressed by an individual about existing disease activity measures not properly addressing patient’s disease activity in the feet were considered and as new disease activity measures are developed, these can also be considered for inclusion in the framework.
DISCUSSION
The present work describes the development of a comprehensive framework for measuring, monitoring, and optimizing RA care in Canada. This work builds upon the legacy of the AAC’s System-level PMs for Inflammatory Arthritis. The PMs selected for inclusion in the current framework were derived from published measures to ensure comparability with other healthcare contexts and included the existing 6 AAC PMs to encourage ongoing monitoring of early access to care and treatment. In contrast to the previous AAC set, which included some measures applicable to other types of inflammatory arthritis, the present framework focuses only on RA. Importantly, the currently proposed RA framework expands measurement beyond the originally proposed AAC measures to evaluate other important processes of care and patient outcomes.
New to the present framework is the inclusion of proposed outcome measures for RA. It should be noted that further operationalization of these and other included measures is required prior to use. Our previous examination of the AAC measures using different data sources revealed data availability to be a critical factor affecting measure operationalization and reporting13. Further, measure results were found to differ depending on the population examined (e.g., RA patients under current rheumatologist care compared to those not currently under active rheumatologist care)14. The highest performance on the measures is often observed in patients participating in longitudinal cohorts, which may not reflect all patients seen in usual practice17. The present measurement set is, therefore, appropriate for quality improvement and research and may not be appropriate for high-stakes measurement (e.g., accreditation or pay for performance programs) as there are further crucial considerations needed around the measurement and reporting of outcomes including appropriate risk adjustment strategies28.
A challenge identified through this process was a lack of existing and feasible PMs addressing some of the framework’s strategic objectives. Namely, those addressing access to multidisciplinary healthcare providers with training and expertise in the assessment and management of RA and the measurement and optimization of patient experience and satisfaction with care. For both of these strategic objectives, there were concerns about feasibility of measurement. From our systematic review, few existing PMs readily addressed the concept of access to multidisciplinary care leading to the proposal of a structure measure capturing the multidisciplinary care workforce, analogous to the existing AAC rheumatology workforce measure. The Delphi panelists felt there would be challenges with measuring, given lack of available data, especially as multidisciplinary care may occur in private or publicly funded settings. A recent survey of advanced practice/extended role practitioners in arthritis and musculoskeletal care highlights the challenges of measuring this workforce given different training, roles, and funding models29. Patient experience with care was also a strategic objective with no PM selected. An adaptation of a measure addressing processes of care important to the patient experience of care was suggested to the Delphi panel because it had previously been published to measure arthritis patient experience with centralized intake30. Although the proposed PM was developed using a similar process, the Delphi panelists first recommended some wording modifications during Round 2; however, even with these modifications, by Round 3, panelists were still concerned about feasibility of data collection leading to the exclusion of the measure from the final set. These concerns about the feasibility of PMs addressing these strategic objectives should not preclude any ongoing efforts to measure and improve patient experience of care and access to multidisciplinary care.
Strategic objective 3 of the framework reflects a desire to enhance a patient-centered approach to care by ensuring they have access to the right information at the right time to participate in decision making. Unfortunately, there were limited existing available PMs for consideration to address this objective and the measure selected will require further operationalization to ensure consistent measurement. Efforts to develop tools including decision aids and promote their use and measure the degree to which shared decision making occurs are areas that could be considered for future inclusion in the framework.
While the present framework was developed using a comprehensive and transparent process leveraging existing quality measurement efforts to ensure concordance with international PMs and guidelines, there are a number of limitations to highlight. The framework was centered around a vision and strategic objectives developed through extensive arthritis stakeholder consultation across Canada; however, it is possible that the objectives for quality improvement may vary in different care delivery contexts and in different countries, especially those with high-stakes performance measurement linked to physician payment. Additionally, this work may not be representative of Indigenous health needs and further work may be necessary to understand if the framework is applicable to the delivery of RA care to Indigenous populations. It is possible that a different composition of Delphi panelists could have generated a different measurement set. As discussed above, the framework includes some strategic objectives where few or no PMs were identified as suitable for inclusion due to concerns around data availability. It is also possible that different measurement concepts may be relevant and appropriate to address the strategic objectives beyond those available in the published literature. This should be viewed as an area in need of additional research and evaluation for future development of the framework, and as a dynamic framework that would be updated as new validated measures become available. Further, while all measures address important concepts, it is possible that some rheumatologists may view some comorbidity measures as outside of their scope of practice (e.g., CV or osteoporosis screening); nonetheless, the measures were retained in the framework to promote a comprehensive approach to care. Importantly, the final framework includes 21 measures and it is possible that in some health systems, data may not be available to measure all of the PMs and/or there may not be resources to support measurement and/or quality improvement efforts in all areas. We propose that interested sites could consider adopting a smaller set of measures for local testing and implementation, as appropriate to their specific context and priorities.
Expanding upon the legacy of the AAC’s System-level PMs, this work proposes a comprehensive framework for quality measurement addressing 7 strategic objectives for quality improvement and a central vision for “ensuring patient-centered, high quality care for people living with RA.” The present framework, while thematically comprehensive, will require further operationalization prior to widespread use, and efforts locally and regionally are currently already underway in some provinces to facilitate measurement and improvement efforts. Future iterations of the framework will be undertaken on a regular basis through consultation with the CRA and stakeholders to drive national quality improvement efforts.
ACKNOWLEDGMENT
The authors would like to acknowledge the support from the Canadian Rheumatology Association (CRA) in helping with the public comment platform for this study. As well we would like to acknowledge the CRA and the Arthritis Alliance of Canada (AAC) for supporting this work through updates to their committees and membership.
APPENDIX 1.
The RA Quality Measurement Delphi Panel Group (in alphabetical order). Vandana Ahluwalia, MD, FRCPC, Rheumatologist, William Osler Health System, ON; Pooneh Akhavan, MD, MSc, FRCPC, Rheumatologist, Mount Sinai Hospital, ON; Matthew Cooper, MD, FRCPC, Rheumatologist, Division of Rheumatology, Department of Medicine, University of Calgary; Natasha Kaur Gakhal, MD, MSc, FRCPC, Rheumatologist, Women’s College Hospital, University of Toronto, ON; Carol Hitchon, MD, MSc, FRCPC, Rheumatologist, Department of Internal Medicine, University of Manitoba, MB; Alison M. Hoens, MSc, BSc, Member, Arthritis Patient Advisory Board and Affiliate Knowledge Broker, Arthritis Research Canada, Clinical Professor, Department of Physical Therapy, University of British Columbia, Research Associate, Centre for Health Evaluation & Outcomes Sciences, University of British Columbia, BC; Shirley Lake, MD, MSc, FRCPC, Assistant Professor, University of Toronto, ON; Jean Légaré, Canadian Arthritis Patient Alliance; Katie Lundon, BScPT, MSc, PhD, ACPAC Program, Office of Continuing Professional Development, Faculty of Medicine, University of Toronto, ON; Laura Passalent, BScPT, MHSc, ACPAC, Physiotherapist Practitioner, Arthritis Program, Toronto Western Hospital, Clinician Investigator, Krembil Research Institute, Lecturer, Department of Physical Therapy Faculty of Medicine, University of Toronto, ON; David B. Robinson, MD, MSc, FRCPC, Associate Professor, Max Rady College of Medicine, University of Manitoba, MB; Leslie Soever, BScPT, MSc, ACPAC, Physiotherapist Practitioner, Arthritis Program, Toronto Western Hospital, University Health Network, ON, Clinical Lecturer, Department of Physical Therapy, University of Toronto, ON; Trudy Taylor, MD, FRCPC, Associate Professor, Dalhousie University, Halifax, NS; Marie Westby, PhD, PT Teaching Supervisor, Mary Pack Arthritis Program, Vancouver, BC, Associate Researcher, Centre for Hip Health and Mobility, Vancouver, BC; Elaine A. Yacyshyn, MD, FRCPC, Associate Professor, University of Alberta, Edmonton, AB; Jessica Widdifield, PhD, Assistant Professor, Institute of Health Policy, Management and Evaluation, University of Toronto and Sunnybrook Research Institute, ON; Michel Zummer, MD, FRCPC, Rheumatologist, Maisonneuve-Rosemont Hospital, Montreal, QC, Associate Professor, University of Montréal, Montreal, QC.
Footnotes
Funding for this study was provided by a Project Grant from the Canadian Institutes of Health Research (CIHR) PJT 153265. DAM is supported through a Canada Research Chair in Health Systems and Services Research, and the Arthur J.E. Child Chair in Rheumatology Outcomes Research. DL is supported by the Mary Pack Chair in rheumatology research from The Arthritis Society of Canada and the University of British Columbia. CEB is funded by a Stars Career Development Award (SI2-169745) from the Canadian Institutes of Health Research Institute of Musculoskeletal Health and Arthritis and The Arthritis Society.
The authors declare no conflicts of interest pertaining to this work.
- Accepted for publication August 17, 2020.
- Copyright © 2021 by the Journal of Rheumatology
REFERENCES
ONLINE SUPPLEMENT
Supplementary material accompanies the online version of this article.
