Abstract
Objective. The Patient Reported Outcome Measurement Information System (PROMIS) is a collection of item banks of self-reported health. This study assessed the feasibility and construct validity of using PROMIS instruments in vasculitis.
Methods. Data from a multicenter longitudinal cohort of subjects with systemic vasculitis were used. Instruments from 10 PROMIS item banks were selected with direct involvement of patients. Subjects completed PROMIS instruments using computer adaptive testing (CAT). The Medical Outcomes Study Short Form-36 (SF-36) was also administered. Cross-sectional construct validity was assessed by calculating correlations of PROMIS scores with SF-36 measures and physician and patient global scores for disease activity. Longitudinal construct validity was assessed by correlations of between-visit differences in PROMIS scores with differences in other measures.
Results. During the study period, 973 subjects came for 2306 study visits and the PROMIS collection was completed at 2276 (99%) of visits. The median time needed to complete each PROMIS instrument ranged from 40 to 55 s. PROMIS instruments correlated cross-sectionally with individual scales of the SF-36, most strongly with subscales of the SF-36 addressing the same domain as the PROMIS instrument. For example, PROMIS fatigue correlated with both the physical component score (PCS; r = −0.65) and with the mental component score (MCS; r = −0.54). PROMIS physical function correlated strongly with PCS (r = 0.81) but weakly with MCS (r = 0.29). Weaker correlations were observed longitudinally between change in PROMIS scores with change in PCS and MCS.
Conclusion. Collection of data using CAT PROMIS instruments is feasible among patients with vasculitis and has some cross-sectional and longitudinal construct validity.
The clinical course of patients with systemic vasculitis is often characterized by alternating periods of remission and active disease. Outcome measures of antineutrophil cytoplasmic autoantibody-associated vasculitides and large-vessel vasculitis used in randomized controlled trials are predominantly physician-based1,2 and do not broadly collect information on the patient experience3,4. A standard approach to patient-reported outcomes (PRO) has not been developed for vasculitis. Measures presenting a more accurate picture of the patient perspective of disease burden in vasculitis would improve disease assessment and the selection of treatments that target disease manifestations of importance to patients5.
The Patient Reported Outcomes Measurement Information System (PROMIS) is a system of instruments intended to record all elements of self-reported health6,7. PROMIS uses item response theory (IRT) as an underlying statistical framework that enables the use of computer adaptive testing (CAT) to arrive at a precise estimate of a trait under study with minimal patient burden. CAT items are selected from the entire item bank established for the specific instrument domain being studied, in which additional items are administered to each participant based on their previous response(s). CAT approaches are designed to efficiently arrive at a measure for each participant with minimal floor and ceiling effects. In addition, to remain compatible with nonadaptive settings, short-form (fixed questionnaires) PROMIS instruments in each domain have been developed, typically consisting of 4, 6, or 8 items.
The development of the PROMIS system has led to the application of a number of standardized measures to many disease areas but this has not been done with the vasculitides. The aim of our study was to assess the feasibility and validity of using PROMIS measures to assess the burden of disease in patients with various vasculitides.
MATERIALS AND METHODS
Study oversight
Our study was performed by members of the Vasculitis Working Group within the Outcome Measures in Rheumatology (OMERACT)8 initiative and the investigators of the Vasculitis Clinical Research Consortium (VCRC). The work was performed in concordance with the OMERACT guidelines of outcome measure development and validation9. Members of the Steering Committee for our study were drawn from North America and Europe and included clinical investigators, methodologists, and patient research partners (PRP). The main center’s ethics review was obtained from the Office of Regulatory Affairs at the University of Pennsylvania (approval no. 815673). Ethical approval was also obtained from each participating site: Boston University (H-24994), Cleveland Clinic (05-053), Mayo Clinic (PR05-004051-12), Mount Sinai Hospital Toronto (07-0189-E), St. Joseph’s Healthcare Hamilton (07-2874), the University of Pittsburgh (PRO10090549), and the University of Utah (IRB_00042415). All subjects enrolled in the VCRC Longitudinal Study (VCRC-LS) have provided written consent for their participation.
Data source/setting
The VCRC-LS is an ongoing longitudinal observational cohort funded by the National Institutes of Health, in which participants undergo in-person assessments at either quarterly or annual visits. A total of 6 forms of vasculitis are studied in the VCRC-LS: eosinophilic granulomatosis with polyangiitis (Churg-Strauss, EGPA), giant cell arteritis (GCA), granulomatosis with polyangiitis (GPA), microscopic polyangiitis (MPA), Takayasu arteritis (TA), and polyarteritis nodosa (PAN). Data for our study were collected from participants included in the VCRC-LS derived from study visits between December 2013 and May 2016.
Selection of PROMIS measures
Selection of candidate domains in vasculitis to be represented by a PROMIS item bank was guided by the previous literature, clinical expertise, guidance from the PRP, a review of qualitative data describing the experiences of patients with vasculitis, and through input from the OMERACT community4,10,11,12,13,14,15. Of the 10 PROMIS item bank instruments selected, 4 were identified as being of highest relevance to outcome measurement in vasculitis and administered to all subjects in the VCRC-LS: fatigue, physical function, pain interference, and cognitive function. To limit patient burden, the 6 other item banks considered possibly relevant to vasculitis (sleep disturbance, social participation, sleep-related impairment, anger, social isolation, and anxiety) were studied by randomized assignment of 2 to be consistently administered to each patient so that each of the 6 additional measures would be completed by one-third of the participants (Figure 1). During the first VCRC study visit after PROMIS had been added to regular assessments, subjects were randomized to 1 of the 3 arms by a simple randomization function at the Web server for this study. Participants remained in the arm to which they were originally randomized. All PROMIS instruments are scaled to have a mean of 50 and SD of 10 in the US population, where higher scores equate to more of the concept being measured.
Administration of PROMIS instruments
During scheduled study visits in the VCRC-LS, all participants were asked to complete the PROMIS instruments using an online system presented on a computer or tablet, in addition to the paper-based PRO instruments already included as part of the study; participants had the option of using one of their fingers or a handheld stylus to respond to the tablet-based questions. Participants were randomized to complete sets of 6 PROMIS instruments as outlined above and each patient completed their originally assigned set of instruments at each followup visit. For each item bank, a CAT instrument was first administered to completion. To also test the short-form version of each PROMIS measure, any remaining items on the 4-question short form that had not been chosen by the CAT protocol were also provided to the participants for completion. PROMIS instruments all use a 7-day recall period, except the physical functioning instrument, in which recall periods are not associated with the items. Initially, a pain screening question was used so that only those patients who reported having experienced pain during the prior week were administered the instruments from the pain interference item banks. Because this use of a screening question resulted in skewed distribution of pain interference scores, the approach was discontinued during the observation period and the PROMIS pain interference instrument was subsequently administered to all participants. For the CAT instrument, the default IRT calibration settings at assessmentcenter.net were used.
Additional measurement of health-related quality of life (HRQOL)
HRQOL was assessed with the Medical Outcomes Study Short Form-36 (SF-36) administered by paper (as most instruments in the VCRC) at all study visits. The SF-36 contains 36 items that assess HRQOL in 8 health dimensions: physical functioning, role physical, bodily pain, general health, vitality, social functioning, role emotional, and mental health16,17. Scores for each dimension/subscale range from 0 to 100 and were transformed to have a mean of 50 and SD of 10 in the reference population, with higher scores indicating better HRQOL. Following standard procedures, 2 summary scores were derived from combining the 8 subscales: the physical component summary (PCS) score and the mental component summary (MCS) score. The PCS and MCS are also norm-based scores standardized to the US general population and transformed to have a mean of 50 and SD of 10 in the reference population.
Assessment of disease activity
Physicians completed a global assessment on a 0–10 integer scale, worded as follows: “Mark to indicate the amount of disease activity (not including longstanding damage) within the previous 28 days.” Patients completed a global assessment on a 0–10 integer scale, worded as follows: “Please mark to indicate how severe your vasculitis disease has been for the past 28 days.” Active disease was defined as a score > 0 on the PGA for disease activity.
Statistical analysis
Means with SD and proportions were used to summarize demographic and disease-associated factors in this cohort for continuous and categorical variables, respectively. Feasibility of PROMIS administration was assessed by calculating the proportion of participants who completed the PROMIS assessments and the time it took to complete the assessments. Cross-sectional construct validity was assessed at the first patient visit during the study period by calculating Spearman correlation coefficients between each PROMIS instrument and subscales of the SF-36, PCS, MCS, and physician’s (PGA) and patient’s global assessments (PtGA). Longitudinal construct validity was assessed by calculating the between-visit difference in PROMIS measures with differences in the SF-36 measures and global scores to evaluate whether the PROMIS measure was more highly correlated with subscales designed to measure the same construct. Data interpreted as supporting construct validity included higher correlations between related measures, convergence (e.g., comparing correlation of bodily pain by SF-36 with pain interference by PROMIS, and comparison of vitality by SF-36 with fatigue by PROMIS), and divergence (lower correlations between measures of unrelated constructs). All statistical analyses were conducted using SAS statistical software version 9.4 (SAS Institute Inc.).
RESULTS
Baseline characteristics
During the study period, 1040 patients with systemic vasculitis came for a VCRC-LS study visit, of whom 973 (94%) started a PROMIS assessment and thus entered our study (Table 1). During the study period the 973 participants attended 2306 study visits and PROMIS was completed at 2276 (99%) of the visits. Of the participants, 62% were female, and the mean age was 57.1 years. Participants with all 6 forms of vasculitis under study participated in the PROMIS assessments: EGPA (n = 123), GCA (n = 194), GPA (n = 433), MPA (n = 73), PAN (n = 53), and TA (n = 97). Randomization of the 6 secondary measures (Figure 1) resulted in study arms that were balanced regarding age, sex, type of vasculitis, and measures of HRQOL (Table 1).
Scores on measures of HRQOL and PROMIS measures were overall reduced compared to US population norms (Table 1). At baseline, PROMIS measures were worse (i.e., higher) than the population norms for fatigue, pain interference, and sleep-related impairment, and worse (i.e., lower) for physical function and social isolation. Measures for cognitive abilities, social participation, anger, and anxiety were close to the population norms. No substantial differences in PROMIS scores across disease categories were identified (Table 2).
Time to completion of PROMIS assessments
The median times to complete each PROMIS domain ranged from 38 to 53 s (Table 3). Age was strongly associated with how long it took subjects to complete the PROMIS assessment. Typically, it took those above the age of 80 years twice as long to complete the assessments compared to those 40 years old and younger. It took participants longer to complete the assessment on the first visit than at subsequent study visits (data not shown).
Cross-sectional validity of PROMIS instruments at baseline
PROMIS instruments correlated highly, and significantly, with the individual scales of the SF-36. Correlations were of varying magnitude and in the expected direction. PROMIS instruments correlated most strongly with subscales of the SF-36 addressing the same domain (Table 4).
Some of the PROMIS instruments correlated moderately or strongly with summary scores of the SF-36 in a pattern consistent with the convergence and divergence expected from these measures. For example, PROMIS fatigue correlated both with PCS (r = −0.65) and with MCS (r = −0.54). PROMIS physical function correlated strongly with PCS (r = 0.81) but weakly with MCS (r = 0.29). PROMIS social participation correlated strongly with PCS (r = 0.72) and moderately with MCS (r = 0.48) and social function subscales. Sleep disturbance and sleep-related impairment correlated best with the SF-36 vitality, social functioning, and MCS scores.
Some PROMIS instruments correlated moderately with the PtGA for disease activity: fatigue, physical function, pain interference, and social participation, all with r > 0.40. Weak or no correlations were observed between PROMIS instrument and the PGA for disease activity (Table 4). Correlation coefficients using PROMIS scores from short forms were almost identical to the coefficients when the corresponding PROMIS CAT scores were used (Supplementary Table 1, available with the online version of this article).
Longitudinal construct validity of PROMIS instruments
Among the 575 participants who came for more than 1 visit, the change in PROMIS scores correlated with change in the validating constructs, typically in the range r = 0.3–0.5 (Table 5). CAT and short-form instruments provided quite similar correlation coefficients (Supplementary Table 2, available with the online version of this article).
DISCUSSION
Our study demonstrates that the administration of PROMIS instruments is quite feasible and valid for use among patients with various forms of vasculitis. Adding PROMIS measures to other disease assessment instruments did not prevent a high completion rate. The completion of each PROMIS instrument took < 1 min for most subjects, allowing for comprehensive assessment with minimal subject burden. PROMIS measures demonstrated cross-sectional construct validity regarding SF-36 domains. Change over time in PROMIS measures also appeared to correlate appropriately with differences in SF-36 measures.
Currently used physician-based measures of disease in vasculitis may not incorporate the effect of symptoms important to patients. Our findings support that hypothesis. The current measures are dichotomized to active or nonactive disease. By formally using the PROMIS system in the evaluation of potential novel treatments for vasculitis, treatment-related benefits of importance to patients are more likely to be recorded. PROMIS measures might also contribute to future definitions of intermediate disease states that cannot be defined with current measures.
Most PROMIS instruments correlated moderately with PtGA for disease activity, with the strongest correlations observed for social participation, fatigue, and pain interference. PtGA has been shown to record a disease aspect that is different from physician measures and it has some attractive characteristics as an outcome measure18,19. The underlying constructs of PtGA in vasculitis are not known, but this analysis offers some insight into which domains of self-reported health contribute to PtGA for disease activity in vasculitis.
None of the 10 PROMIS instruments correlated with the PGA, highlighting that physicians and patients bring different perspectives to the assessment of vasculitis13,20 and that information on the life effect of vasculitis must be directly assessed by patients. This is consistent with what has been observed in other rheumatic diseases: that physician-based and patient-based measures record distinct disease features21,22,23. To assess how therapeutic strategies could diminish the effect of disease, PRO should be administered as part of randomized controlled trials.
Within each domain, the distributions of the scores from the PROMIS CAT instrument and the short form were almost identical. Correlation coefficients between PROMIS and other measures were also quite similar between the CAT and short-form instruments. Future studies may show a degree of separation between the 2 PROMIS formats and determine whether such a difference is meaningful. Therefore, both CAT and short forms are reasonable to use for disease assessment in vasculitis.
There is a clear mandate to formally incorporate the patient perspective into assessment of vasculitis. We found that instruments from several domains might be of importance. That is informative but does not imply that all the studied domains require representation in outcome measures administered in a trial. The goal should be to arrive at a parsimonious set of instruments that record the effect of disease and allow for discrimination between treatment arms. We identified several domains that might be fit for this purpose. Further studies might identify the optimal items and CAT algorithms for assessment of vasculitis. It is also possible that administration of PROMIS instruments could be feasible in clinical care to obtain assessment of life effects of vasculitis without exhaustively burdening the patient with forms to fill in.
Our study had several important strengths. The data source is a well-defined patient cohort with systemic vasculitis followed in a longitudinal fashion at expert centers according to a protocol that provides for evaluations of changes in HRQOL and changes in physician-based disease assessment during the disease course. The large number of patients participating and the comprehensive data acquisition is also notable. The domains under study were chosen with substantial patient input to reflect disease-related manifestations of importance to patients with vasculitis.
Our study also has some limitations to consider. First, as is common in cohorts of patients with vasculitis, participants were in disease remission about 80% of the time. Thus, it remains to be seen how the identified measures will function in trials that enroll patients during a period of highly active disease. Second, the different PRO instruments in our study used different recall periods that might have affected our findings. Third, there are no qualitative data or surveys on how patients with vasculitis felt the PROMIS instruments recorded the disease burden of vasculitis. Fourth, our study was conducted in a North American cohort of vasculitis patients and the results might not be generalizable to other cohorts. Translation of most of the PROMIS instruments is under way but validation in non–English-speaking cohorts is warranted.
Our study demonstrates the validity of using PROMIS instruments in vasculitis, and the feasibility and construct validity of several PROMIS instruments with systemic vasculitis. It also expands the options for recording PRO in the assessment of the burden of vasculitis.
ONLINE SUPPLEMENT
Supplementary material accompanies the online version of this article.
Footnotes
This work was supported by The Vasculitis Clinical Research Consortium (VCRC; U54 AR057319 and R01 AR 064153), which is part of the Rare Diseases Clinical Research Network, an initiative of the US Office of Rare Diseases Research, National Center for Advancing Translational Science (NCATS). The VCRC is funded through collaboration between NCATS, and the National Institute of Arthritis and Musculoskeletal and Skin Diseases, and has received funding from the National Center for Research Resources (U54 RR019497). Additional support for this work was provided by a contract from the Patient-Centered Outcomes Research Institute (IP2PI000603).
- Accepted for publication November 26, 2018.