Abstract
Objective. The antineutrophil cytoplasmic antibody–associated vasculitides (AAV) are multiorgan diseases. Patients with AAV report impairment in their health-related quality of life (HRQOL) and have different priorities regarding disease assessment compared with physicians. The Outcome Measures in Rheumatology (OMERACT) Vasculitis Working Group previously received endorsement for a core set of domains in AAV. Two approaches to measure patient-reported outcomes (PRO) were presented at OMERACT 2016.
Methods. A novel 5-step tool was used to facilitate assessment of the instruments by delegates: the OMERACT Filter 2.0 Instrument Selection Algorithm, with a red-amber-green checklist of questions, including (1) good match with domain (face and content validity), (2) feasibility, (3) do numeric scores make sense (construct validity)?, (4) overall ratings of discrimination, and (5) can individual thresholds of meaning be defined? Delegates gave an overall endorsement. Three generic Patient-Reported Outcomes Measurement Information System (PROMIS) instruments (fatigue, physical functioning, and pain interference) and a disease-specific PRO, the AAV-PRO (6 domains related to symptoms and HRQOL), were presented.
Results. OMERACT delegates endorsed the use of the PROMIS instruments for fatigue, physical functioning, and pain interference (87.6% overall endorsement) and the disease-specific AAV-PRO instrument (89.4% overall endorsement).
Conclusion. The OMERACT Vasculitis Working Group gained endorsement by OMERACT for use of the PROMIS and the AAV-PRO in clinical trials of vasculitis. These instruments are complementary to each other. The PROMIS and the AAV-PRO need further work to assess their utility in longitudinal settings, including their ability to discriminate between treatments of varying efficacy in the setting of a randomized controlled trial.
Antineutrophil cytoplasmic antibody–associated vasculitis (AAV) consists of 3 multisystem diseases caused by inflammation of the small blood vessels: granulomatosis with polyangiitis, eosinophilic granulomatosis with polyangiitis (Churg-Strauss), and microscopic polyangiitis. Because of their relative rarity and overlapping disease features, these vasculitides are commonly studied together within randomized controlled trials (RCT)1. Modern therapeutic regimens, including high-dose glucocorticoids and immunosuppressive medications, have transformed AAV from a nearly universally fatal disease to a usually treatable problem2. However, patients still often experience persistent and/or relapsing disease and irreversible damage3 from the effects of both the disease manifestations and the toxicities of treatments4.
From the onset of disease in AAV, patients’ health-related quality of life (HRQOL) is impaired5. There is a discrepancy between the perspectives of patients with AAV, who rank constitutional symptoms such as fatigue/reduced energy levels as having the greatest relevance to their disease, and that of their physicians, who rank the effects of organ damage such as requirement for renal replacement therapy or oxygen dependence as being of greater importance6. Therefore, it is essential to collect patient-reported outcomes (PRO) within clinical trials of new treatment regimens to ensure that outcomes of importance to patients are accurately measured7.
Generic HRQOL instruments, such as the Medical Outcomes Study Short Form-36 (SF-36) or the EQ-5D, can be applied in a range of different disease populations and interventions and facilitate comparisons between both diseased and general populations8. However, these tools may not be specific enough to identify the complexity of experiences of patients within particular diseases. Disease-specific instruments, for example the Rheumatoid Arthritis Impact of Disease score9 or the Bristol Rheumatoid Arthritis Fatigue Multi-Dimensional Questionnaire10, may perform better at identifying such experiences. It is generally recommended that both generic measures and disease-specific PRO be used to provide a comprehensive and relevant description of any individual population11.
In 2010, the Outcome Measures in Rheumatology (OMERACT) Vasculitis Working Group received endorsement for a core set of domains and outcome measures for use in clinical trials in AAV12. Within the “patient-reported outcome” domain, the SF-36 was presented as the generic instrument for use in AAV12. The lack of a disease-specific PRO and the relative lack of research into PRO in vasculitis were noted12. The SF-36 was included in the core set because it can discriminate between disease states of importance in AAV, i.e., remission versus active disease, and its scores correlate moderately well with disease activity, as measured by the clinician-completed Birmingham Vasculitis Activity Score/WG13. However, there have been concerns that the SF-36 does not sufficiently identify specific disease manifestations identified by patients with AAV as being important6,14,15.
The OMERACT Vasculitis Working Group established a strategy to analyze the patient perspective in more depth, and to develop and/or validate new PRO for use in clinical trials of AAV. This strategy has been facilitated through workshops held at the 2012 and 2014 OMERACT conferences, 2 face-to-face meetings in the United States and United Kingdom, and monthly teleconferences with an international Steering Committee of patient partners, qualitative and quantitative methodologists, and clinician investigators16.The 3 Vasculitis Working Group projects are:
Analysis of the utility of domains of the Patient-Reported Outcomes Measurement Information System (PROMIS)17 for use in AAV;
Development and validation of a disease-specific PRO for use in AAV (AAV-PRO); and
Analysis and application of the International Classification of Function, Disability and Health (ICF) in AAV.
At OMERACT 2016, the Vasculitis Working Group presented the development and validation steps performed for the PROMIS instruments (individual domains of fatigue, physical functioning, and pain interference) and an AAV-PRO. OMERACT delegates were asked to assess each instrument by use of a novel 5-step assessment tool, the OMERACT Filter 2.0 Instrument Selection Algorithm (OFISA) and red-amber-green checklist (also known as the “eyeball test”; Figure 1). These 5 steps assess whether the instruments are a good match with the domain (face and content validity), feasibility (practicability, length, burden, cost, access, and translations), whether the numeric scores make sense (construct validity), whether the instrument discriminates between different states and in situations of change, and whether thresholds of meaning are defined. During the workshop, delegates voted to determine whether each of the 5 steps was achieved. During the final meeting, plenary session delegates voted to approve or decline each of the steps individually, and then to provide an overall endorsement of each instrument. The eyeball test is closely related to the OMERACT Checklist for Instrument Selection for Core Outcome Measurement Sets, which was also fulfilled for the instruments under study (Figure 2). Breakout groups during the workshop allowed for greater scrutiny of each instrument, analysis of next steps needed for their development, and facilitation of additional discussion around the use of the ICF in AAV. Feedback received from the OMERACT community on the 3 projects is given in the sections below.
The OMERACT Filter 2.0 Instrument Selection Algorithm (OFISA) and red-amber-green checklist. Adapted with permission from the OMERACT Handbook. OMERACT: Outcome Measures in Rheumatology; R: red; A: amber; G: green.
Full evaluation of Core Outcome Measurement Sets is to be completed (5.D.1–12; The OMERACT Handbook: www.omeract.org/pdf/OMERACT_Handbook.pdf). OMERACT master checklist for developing Core Outcome Measurement Sets. OMERACT: Outcome Measures in Rheumatology; ANCA: antineutrophil cytoplasmic antibody; PROMIS: Patient-Reported Outcomes Measurement Information System; OFISA: OMERACT Filter 2.0 Instrument Selection Algorithm; RCT: randomized controlled trial; AAV-PRO: patient-reported outcome measure for ANCA–associated vasculitis.
Our OMERACT report includes a summary of the development and validation of both general and disease-specific PRO in AAV, all of which is novel for this field and substantially advances outcome research in vasculitis. The specifics of the several component projects will be published separately.
PROMIS Instruments for Fatigue, Physical Functioning, and Pain Interference in AAV
The PROMIS is a generic item bank intended to cover all aspects of self-reported health17. Physical functioning is a core domain in measurement of disease effect among patients with rheumatic diseases, and fatigue and pain are consistently ranked as important disease manifestations among patients with AAV6. The PROMIS can be administered by computer adaptive testing (CAT) and on paper as short-forms that typically include 4, 6, or 8 questions. Administration by CAT could result in increased precision, but requires access to a computer. The PROMIS has dedicated instruments to measure these domains and might have 2 particularly attractive qualities for use in RCT for AAV. (1) PROMIS measures are precise, which can lead to greater power to examine subgroups. This is important for this multisystem disease in which several subgroups have been identified based on different organ manifestations. Increased precision also helps the conduct of smaller RCT or to detect smaller differences in treatment efficacy, important characteristics for any disease, but especially for a rare disease. (2) PROMIS measures are intended to be responsive (sensitive to change). AAV is often characterized by fluctuating levels of disease activity, therefore responsiveness (i.e., sensitivity to change) is a key positive feature for an outcome measure for use in longitudinal studies and RCT in AAV.
At OMERACT 2016, data pertaining to content validity, construct validity, and responsiveness were presented, discussed, and endorsed by the delegates in relation to the OFISA eyeball test (Green-level endorsement; Figure 1, Figure 2, and Figure 3). It was also decided that numerical scores were relevant for individual patients and could discriminate between disease states of importance in AAV (Amber-level endorsement). Data were collected through CAT, but also included all the items that are administered by the 4-question short forms. We received feedback from some OMERACT delegates that the feasibility of the PROMIS by CAT has not been assessed in the setting of RCT and we interpret that endorsement of the feasibility PROMIS pertains to its administration by short-forms or CAT. The next step will be to administer these PROMIS instruments in the setting of an RCT to assess whether PROMIS instruments discriminate between treatment arms.
OMERACT endorsement of PRO in AAV. (A) PROMIS instruments for fatigue, physical functioning, and pain interference for AAV voting results. (B) The disease-specific AAV-PRO voting results. OMERACT endorsement set at ≥ 70% of votes (sum of Green or Amber). Green (OK): Yes, I agree; Amber (OK): I am okay with this, but have some reservations, more work needed in this area; Red (Not OK): I disagree; Grey: Insufficient evidence or information. OMERACT: Outcome Measures in Rheumatology; PRO: patient-reported outcomes; ANCA: antineutrophil cytoplasmic antibody; AAV: ANCA–associated vasculitis; PROMIS: Patient-Reported Outcomes Measurement Information System; AAV-PRO: PRO measure for AAV.
The Disease-specific AAV-PRO
The AAV-PRO is a disease-specific PRO instrument for AAV. It is a profile instrument consisting of 29 items representing 6 domains: organ-specific symptoms, systemic symptoms, treatment side effects, social and emotional effect, concerns about the future, and physical function.
The results of a comprehensive program of qualitative research were presented at OMERACT, including 50 in-depth individual interviews from the United Kingdom, United States, and Canada, and 2 focus groups in the United States, all of which were used to identify themes of importance to patients with AAV. Themes identified were recast as candidate items, which then underwent extensive review, piloting, cognitive testing, and linguistic and translatability assessments. Our work provided evidence that the AAV-PRO was a good match with the domain of PRO and was feasible to use. OMERACT delegates endorsed steps 1 and 2 of the OFISA eyeball test questions at the Green level (Figure 3).
The initially developed long-form (35 candidate item) AAV-PRO questionnaire has undergone large-scale testing among patients with AAV to inform item reduction (yielding the final 29-item questionnaire) and to assess scale and measurement properties. This exercise included a test-retest exercise and 3-month followup survey with transition questions. These data were presented at OMERACT and delegates voted to endorse step 3 of the OFISA eyeball test questions at the Green level, and steps 4 and 5 at the Amber level (Figure 3). At the final plenary, OMERACT delegates voted to endorse the AAV-PRO at the eyeball level (89% agreement to endorse; prespecified OMERACT endorsement level was ≥ 70% of votes; Figure 3).
Feedback from 2 breakout groups recommended that the AAV-PRO should be tested next in a cohort of patients likely to exhibit greater change in their disease state over a longer time period to better define thresholds of change that are meaningful to patients. To gain further insights into its construct validity, the AAV-PRO should also be tested against other instruments, such as clinician-derived measures of disease activity and other symptom-specific and generic PRO. Another area of discussion was the scoring of the AAV-PRO. The AAV-PRO is a multidimensional instrument with each separate domain having good internal consistency and consistent with the polytomous Rasch model18. Therefore, each domain can be scored separately. However, clinicians may be keen to create a more pragmatic scoring method for the AAV-PRO. Opinions varied within the breakout groups, but there was consensus that at present all domain scores should be recorded separately; future work could examine use of combining domains into 1 or more summary scores, or identify and concentrate on specific domains of interest within individual trials. An example of where the treatment-related adverse effects domain would be of particular interest could be an RCT of a glucocorticoid-sparing agent.
ICF in AAV
The ICF was endorsed as a health status framework and a classification system for standardized description of an individual’s health and disability by the World Health Organization19. Since then it has found many applications, including endorsement by the OMERACT of ICF as a tool to identify and describe domains relevant to outcome measurement for a specific medical condition20.
The OMERACT Vasculitis Working group is analyzing the ICF, first as a tool to refine the list of domains included in the current OMERACT core set for AAV by identifying domains (described using the ICF “categories”) of importance to specific stakeholder groups, as recommended by the OMERACT Filter 2.0 framework20. Completed steps of this process include (1) identifying ICF categories (each representing a domain) sampled by instruments used in clinical trials of AAV21, (2) identifying domains most relevant to patients through individual interviews (in collaboration with AAV-PRO project described above) followed by a prioritization exercise, and (3) identifying domains prioritized by clinicians with expertise in vasculitis22. Second, the ICF could be used to identify potential contextual factors, which might modify outcome assessment.
One of the breakout sessions of the workshop focused on discussing the results and implications of the ICF-related studies described above and the future directions of this research. The ultimate goal of this initiative is to develop ICF core sets for AAV, a selection of ICF categories (corresponding to OMERACT domains in Filter 2.020) relevant to the study of AAV23. The ICF core sets for AAV would complement and refine the existing OMERACT core set of domains for AAV12.
Summary
The generic PROMIS instruments (for fatigue, physical functioning, and pain interference) and the AAV-PRO (disease-specific PRO for AAV) have been carefully assessed by OMERACT delegates, including patient partners, methodologists, clinician researchers, representatives of the pharmaceutical industry, and regulatory advisers, and have been endorsed at the OFISA eyeball test level. Future work for both projects will complete the final validation steps required per the OMERACT process, including additional longitudinal analysis in cohorts of patients exhibiting greater change in disease state over longer time periods to calculate minimal clinically important differences with greater accuracy. Additional comparisons with other outcome measures will more comprehensively examine different aspects of construct validity of these instruments for use in vasculitis. The ICF project will now compile the results of the 3 completed studies and develop the ICF Core Sets for AAV. The ICF Core Sets will complement and help refine the existing core set of outcome domains for AAV. These 3 projects are complementary and have benefited from a common Steering Committee that includes patient partners, and critical review through the OMERACT process. Each project will continue to be supported and advanced by the OMERACT Vasculitis Working Group.
Acknowledgment
The Outcome Measures in Rheumatology (OMERACT) Vasculitis Working Group appreciates the efforts and interest of all the attendees at OMERACT 2016 who participated in the Vasculitis Workshop, and especially Dr. Dorcas Beaton, Dr. Sarah Hewlett, Dr. Catherine Hill, Dr. Sarah Mackie, and Dr. Alexa Meara.
Footnotes
Supported through 2 Patient Centered Outcomes Research Institute Awards: Contract Number IP2-PI000603 for Pilot Award and Contract Number PPRN-1306-04758 for the Vasculitis Patient-Powered Research Network (part of the Patient-Centered Outcomes Research Network). The Vasculitis Clinical Research Consortium (VCRC; U54 AR057319, U01 AR5187404, and R01 AR064153) is part of the Rare Diseases Clinical Research Network, an initiative of the Office of Rare Diseases Research, and the US National Center for Advancing Translational Science (NCATS). The VCRC is funded through collaboration between NCATS and the US National Institute of Arthritis and Musculoskeletal and Skin Diseases, and has received funding from the National Center for Research Resources (U54 RR019497). UK funding is from the Medical Research Fund, Oxford, and the Oxfordshire Health Services Research Committee, Oxford. Patient survey support is from Vasculitis UK. Dr. Robson was supported by a UK National Institute for Health Research Academic Clinical Lectureship. Dr. Milman was supported by a UCB/Canadian Rheumatology Association/Arthritis Society postgraduate rheumatology fellowship award and a research fellowship from the Department of Medicine at the Ottawa Hospital.
- Accepted for publication January 13, 2017.
REFERENCES
In this issue
