Abstract
Objective. The Childhood Health Assessment Questionnaire (CHAQ), though widely used for assessments in pediatric rheumatology, has drawbacks, including low correlation to disease activity and ceiling effects. We sought to determine if any tools from the Patient Reported Outcomes Measurement Information System (PROMIS) improve on these shortcomings and/or are preferred by patients.
Methods. Patients 5–17 years of age with juvenile idiopathic arthritis (JIA) or juvenile dermatomyositis (JDM) were recruited from the rheumatology clinics at a Canadian children’s hospital. Participants completed the CHAQ, 3 PROMIS measures (pain interference, mobility, and physical activity), and underwent a standard clinical assessment.
Results. Fifty-two patients participated, 25 with JIA and 27 with JDM. None of the PROMIS measures suffered from ceiling effects, whereas the CHAQ Disability Index (DI) and pain visual analog scales both did, with 50% and 20% of patients achieving the best possible scores, respectively. The PROMIS mobility was moderately correlated to the CHAQ-DI (rs –0.60, 95% CI –0.75 to –0.40), and the PROMIS pain interference was strongly correlated to the CHAQ pain score (rs 0.65, 95% CI 0.43–0.80). No measures correlated with disease activity. Patients preferred the PROMIS to the CHAQ.
Conclusion. The PROMIS pain interference, mobility, and physical activity measures improve in some areas where the CHAQ is weak: they do not suffer from ceiling effects, and patients prefer the PROMIS tools. More work is needed to determine the correlation and responsiveness of the PROMIS tools to changes in disease activity over time before they should be widely adopted for clinical use.
- disease activity score
- health assessment questionnaire
- juvenile idiopathic arthritis
- outcomes
- pediatric dermatomyositis/polymyositis
Juvenile idiopathic arthritis (JIA) and juvenile dermatomyositis (JDM) are 2 chronic rheumatic diseases of childhood that, untreated, lead to activity limitation and participation restriction.1,2
In order to monitor treatment, it is important to evaluate physical function deficits (activity limitation and participation restriction) for both of these conditions. In fact, physical functional ability is part of the core sets used for both of these conditions.3,4 The Childhood Health Assessment Questionnaire (CHAQ) is specifically mentioned as part of the core set for JDM; while no measures are mentioned specifically as part of the core set for JIA, the CHAQ is the most commonly used measure of functional status.3,5
The CHAQ is a validated and highly cited patient-reported outcome (PRO), and is administered to all patients with JIA and JDM followed in the rheumatology clinic at The Hospital for Sick Children (SickKids) as part of the standard of care.
However, the CHAQ suffers from a few drawbacks that may limit usefulness. These include poor correlation to disease activity (as measured by the Disease Activity Score [DAS] for patients with JDM6 and the Juvenile Arthritis Disease Activity Score [JADAS] for patients with JIA7) during low disease activity, poor correlation to physician global assessments, poor responses to changes in disease activity, and a marked ceiling effect.8–14 (A ceiling effect is said to be present if 15% or more results achieve the best score possible.15) Further, the CHAQ is relatively long, may be tedious to complete, and is somewhat complicated to score. These limitations might affect its effectiveness as a tool to evaluate patient-reported disease activity over time.
A potential alternative is the Patient Reported Outcomes Measurement Information System (PROMIS). PROMIS is a collection of PRO tools that can utilize computer-adapted technology and are built on item response theory (IRT), which calibrates the best items to include based on the frequency with which they are chosen and their ability to distinguish between patients.16 IRT allows for the development of effective and efficient tools. PROMIS tools are validated and standardized for use across numerous medical conditions, including rheumatoid arthritis, and are calibrated to be more normally distributed across all levels of disease activity, relative to a standard reference population.17,18,19
PROMIS tools cover a wide range of topics, some of which measure domains comparable to those covered by the CHAQ. These include the PROMIS measures of pain interference, physical activity, and physical function–mobility.
Since PROs are considered an integral part of assessment of disease activity and patient care, it is important to use the most reliable, accurate, and convenient one(s) possible.20,21
While previous studies have independently shown PROMIS to improve in the aforementioned areas in which the CHAQ is weak, to the best of our knowledge there are no studies that directly compare the PROs in both patients with JIA or JDM, making it difficult to determine if 1 PRO significantly out-performs the other in the same cohort.
Considering that the CHAQ has some drawbacks that could compromise its function in clinical practice, alternatives should be studied using comparative methods within the same patient population.
We therefore asked the following research questions: (1) Are the PROMIS pain interference, physical activity, and physical function scores correlated to the CHAQ? (2) Do the selected PROMIS measures better correlate to disease activity in patients with JIA and JDM than the CHAQ? (3) Does the PROMIS exhibit less of a ceiling/floor effect than the CHAQ in patients with JIA and JDM? and (4) Which PRO do patients prefer?
METHODS
Participants. We conducted our study at the SickKids rheumatology clinic. All patients with a diagnosis of JIA or JDM between the ages of 5–17 years were eligible. Patients were excluded if they were new (first clinic visit) or were not proficient enough in English to answer the questionnaires. The study was approved by the institutional research ethics board (protocol reference number: 1000061960); participants and/or their parents or guardians gave written informed consent or verbal assent.
Study procedures. Each patient completed an electronic version of the CHAQ and 3 PROMIS computer adaptive tests (physical function– mobility, pain interference, and physical activity). The order of the PRO administration was randomized, using a random number generator (random.org) to avoid order effects.22 A third survey was provided to determine which PRO the patient preferred, and comprised 4 questions using a 5-point Likert scale ranging from strongly agree to strongly disagree, an overall preference statement, and a text box giving patients the opportunity to explain their answers. To ensure data on preference concerned content only, both the CHAQ and PROMIS computer adaptive tests (CATs) were completed on the same device (either a laptop or tablet) using the REDCap platform hosted at SickKids.23,24 Patients who were unable to complete the questionnaires on their own received assistance from their parent/guardian or a study team member. For patients aged 5–8 years, parent proxy versions of the PROMIS tools were used.
All other data used for the study are collected as part of the current standard of care for these patients and were extracted from the patients’ medical records following the clinic visit. These included the information required to calculate disease activity scores and basic demographic information.
Study measures. For patients with JIA, disease activity was calculated using the clinical Juvenile Arthritis Disease Activity Score (cJADAS10). This abridged tool was developed based on the JADAS, which comprises 4 elements: an active joint count, a physician global assessment, a patient visual analog scale (VAS) of well-being, and the erythrocyte sedimentation rate (ESR).7 Unlike the JADAS, the cJADAS10 does not incorporate the ESR, and limits the active joint count to 10. The cJADAS10 is scored from 0 (no disease activity) to 30 (maximal disease activity).25
For patients with JDM, disease activity was calculated using the DAS, which measures clinical indicators of JDM pertaining to both muscle and skin disease severity, and yields a score between 0–20, with a higher score indicating higher disease activity.6
All patients completed the CHAQ, which consists of 30 questions covering 8 domains: dressing and grooming, arising, eating, walking, hygiene, reach, grip, and activities; these constitute the disability index (DI). The CHAQ-DI score ranges from 0 (no disability) to 3 (severe disability), with a score of at least 0.75 indicating clinically significant deterioration. To measure pain, the CHAQ also incorporates a 100-mm VAS.26
We used the CAT versions of the 3 selected PROMIS measures. The CAT provides a t-score with associated standard error. For each measure, a score of 50 represents the mean score of a general population reference sample. For our selected PROMIS measures, a score > 50 represents more pain interference (worse pain),27 better mobility,28 and more physical activity29 than the general reference population.
Sample size. To calculate our required sample size (n), we used a standard table of correlation coefficients.30 Our goal was to choose an n that would be likely to detect a moderate correlation between the CHAQ or the PROMIS, and one of our measures of disease activity in JIA or JDM. Thus, we chose an n of a minimum of 50.
Statistical analysis. We used descriptive statistics (means and SDs, or medians, IQR, and range of values [ROV]) to summarize our cohort. We looked at associations between the CHAQ, PROMIS, and disease activity measurements using the Spearman rank correlation coefficient (rs), because as expected, the CHAQ was not normally distributed. To facilitate comparison across both JDM and JIA, we standardized DAS using z score adjustments. The 95% CIs for Spearman rank correlations were estimated by 1000 bootstrap replicates. Strength of the correlations was interpreted according to the following definitions: ≤ 0.40 indicating poor, > 0.40 to ≤ 0.60 indicating moderate, > 0.60 to ≤ 0.80 indicating strong, and > 0.80 indicating excellent correlation.31 All statistical analysis and data visualization were completed using R version 3.5.1 (R Core Team; www.R-project.org).
RESULTS
Patient cohort. Table 1 shows a summary of our patient cohort; 52 patients were enrolled into the study (mean age 12 yrs, SD 4) where 27 were patients with JDM and 25 were patients with JIA. Our cohort had low levels of disease activity with a median (IQR, ROV) cJADAS10 score of 1.05 (0–2.9, 0–16.1) for the patients with JIA and a median (IQR, ROV) DAS score of 1.0 (0–3.5, 0–13.0) for the patients with JDM (Figure 1).
Distribution of PRO scores and ceiling effects. The distribution of the CHAQ and PROMIS scores were plotted and checked for ceiling effects (Figure 2). The CHAQ-DI exhibited a large ceiling effect with 50% of patients achieving the best possible score (DI = 0; no disability). The CHAQ pain VAS also had a strong ceiling effect with 20% of patients reporting the best possible score (0, no pain). To confirm that the ceiling effects were not simply due to patients with inactive disease, the distributions were reexamined only with patients with DAS scores > 0 or cJADAS10 scores > 1; the ceiling effects of both the CHAQ-DI and CHAQ pain VAS remained (Figure 3). All the PROMIS tools exhibited a wider distribution of scores, visually nearing a normal distribution, with the exception of the PROMIS pain score, which was skewed toward less pain. None of the PROMIS tools exhibited a ceiling effect.
Correlation between measures. The PROMIS mobility tool had a moderate correlation to the CHAQ-DI (rs –0.60, 95% CI –0.75 to –0.40) and the PROMIS pain interference tool had a strong correlation to the CHAQ pain score (rs 0.65, 95% CI 0.43–0.80). The relationship between the PROMIS mobility score and CHAQ-DI was negative, as expected; as mobility increased, disability was reduced. The PROMIS physical activity score was poorly correlated to the CHAQ-DI (rs = –0.21, 95% CI –0.47 to 0.05; data not shown).
None of the PROMIS scores or the CHAQ-DI were correlated with the DAS, the cJADAS10, or the overall adjusted disease activity score. The CHAQ pain score was poorly correlated to the cJADAS10 and the overall adjusted disease activity score, but not to the DAS (Table 2).
Patient preference. Patients indicated a preference for the PROMIS tools (Figure 4). While our patients reported liking the PROs in general, a minority of participants felt the CHAQ was neither easy (9.6%) nor convenient (15.4%) to complete. In contrast, no one indicated the PROMIS tools were difficult or inconvenient to complete. Further, more participants “strongly agreed” with the ease and convenience of the PROMIS compared to the CHAQ (ease: 59.6% vs 44.2%; convenience: 65.4% vs 44.2%).
When asked which PRO they preferred overall, 50% of patients chose the PROMIS tools, while only 10% chose the CHAQ. A proportion responded that they liked both PROs equally (27%), and 14% responded that they had no preference for either PRO (data not shown).
DISCUSSION
In our sample of patients in the rheumatology clinics at SickKids, we found that the PROMIS tools improve in areas where the CHAQ has some limits; the PROMIS tools exhibited no ceiling effects, but like the CHAQ, they have no correlation to disease activity for both JDM and JIA. Further, patients preferred the PROMIS tools over the CHAQ, even when both were administered electronically. Given these findings, it would be prudent to explore the wider adoption of the PROMIS tools; however, more research needs to be done to determine the optimal combination of measures as well as their responsiveness to change in disease status over time.
PROs are an important part of patient care, as they have been shown to improve treatment quality when included in clinical practice to help inform treatment decisions; it would appear to be prudent for clinicians to use the most accurate and convenient PROs available.20,21 The CHAQ is currently used extensively in pediatric rheumatology. However, while it has some weaknesses, a lack of comparative studies makes it difficult to identify viable alternatives.14
Few studies have investigated the relative functionality of the CHAQ and PROMIS in the same cohort. A recent study by Trachtman and colleagues compared the PROMIS, CHAQ, and another JADAS variant, the JADAS-71, in patients with JIA. Similar to our results, the PROMIS physical function domain was strongly correlated to the CHAQ and less strongly correlated to disease activity.32
Our results suggest that the PROMIS pain interference and physical function–mobility tools and the CHAQ perform similarly in assessing the functional status and pain levels of both patients with JIA and JDM. However, the wider distribution of scores on the PROMIS measures (i.e., the lack of a ceiling effect) indicate that these tools may be sensitive to a wider range of disease activity compared to the CHAQ, and might, therefore, better represent disease spectrum in patient conditions. Both the PROMIS pain interference and mobility measures have been shown to be sensitive to change over time across a variety of pediatric conditions, including children with chronic pain and JIA.33,34,35
The PROMIS tools included in this study were selected with the intention of replicating the domains covered by the CHAQ. However, not all the selected PROMIS tools correlated well with the CHAQ, indicating that other PROMIS tools should be tested if the goal is to replace legacy measures with PROMIS. We found the PROMIS physical function–mobility and pain interference tools to have scores that were correlated to the CHAQ, but acknowledge that PROMIS might have less utility in assessing a patient at a specific clinical encounter. The CHAQ was specifically developed to measure functional ability, where a higher score represents a decreased functional ability due to underlying disease activity. The CHAQ contains detailed questions concerning the patient’s ability to perform various activities involving most parts of the body, while the PROMIS mobility tool focuses on mobility in general, and therefore the CHAQ may, at times, be more useful for individual clinical care. Neither of these tools are well correlated with physician-assessed disease activity.
Neither the PROMIS nor the CHAQ exhibited a strong correlation with the JIA and JDM disease activity measures. While it is possible, this means that PROs are a poor descriptor of disease activity, since disease activity is considered a separate construct; it is also possible that a lack of visible relationship was due to our small sample size. Our study was powered to detect moderate correlations with a sample size of at least 50, which was reduced by half in the disease-specific subgroup analysis. Given that these tools aim to measure different constructs, and that none of the tools directly measures disease activity, one might expect a low correlation. However, the PROMIS measures used in this study and the CHAQ do measure constructs that are often affected by disease activity; therefore, we feel it was suitable to assume that they may be at least moderately correlated to disease activity.
The CHAQ was initially developed in the early 1990s to measure the construct of functional ability/disability.26 Currently, the community more often thinks in terms of function rather than disability,36 and it is likely that the functional problems our patients experience are too mild to be captured by the CHAQ.37 The ceiling effects seen in our cohort may not be the fault of the CHAQ, per se; it is just that our patients do not meet the spectrum of what the CHAQ considers a disability. To further examine this idea, we performed a sensitivity analysis looking at the distribution of scores on the CHAQ with the patients with active disease (cJADAS10 score > 1 or DAS score > 0) and found that the ceiling effects persisted. This suggests that functional limitations due to underlying disease activity may be different than those captured by the CHAQ. A modified version of the CHAQ has been previously proposed, with some new questions that expand on the items currently covered in the CHAQ, such as playing team or individual sports, doing activities for a long period of time without getting tired, and completing highly dexterous tasks. These new questions improved the sensitivity of the CHAQ and suffered less from a ceiling effect.37
Clinical functionality aside, patients preferred the PROMIS over the CHAQ. Patients agreed that the PROMIS was quicker, easier, and more convenient to complete as compared to the CHAQ. This is an important consideration, as optimizing efficiency and convenience should be goals when establishing clinic routine, and any opportunity to improve the quality of care and overall clinical experience of pediatric patients should be acknowledged and encouraged. However, given that almost all clinical trials, especially those for JIA, are international, the extent of translation and cultural validation of the tools must also be considered. The CHAQ is available and validated in many languages and countries, making it easy to use in multinational trials.5 The PROMIS instruments, on the other hand, are widely available in English, with limited availability in French, Spanish, German, and Dutch.38 While PROMIS does have a process for developing new translations, additional work will be required to translate and culturally validate the PROMIS tools before widespread adoption can be considered.
Our study must be interpreted in the context of several potential limitations. Our sample size was relatively small and on the lower end of the range required to detect a moderate correlation between measures. Disease-specific subgroup analysis rendered our sample size even smaller for those groups, which may explain why we were unable to show a correlation between the PROMIS tools and disease activity, where others have shown the measures to be responsive to changes in disease activity.33 To address this issue, we standardized our measures of disease activity to allow for analysis of our entire sample as 1 group and still failed to show a relationship. Therefore, it is likely that there is not a strong correlation between these PROs and disease activity. However, given that the cardinal manifestations of JIA (pain and stiffness) and JDM (weakness and rash) differ, we would not expect to see the same degree of correlations across scales in these 2 groups.
It is also possible that we did not see a correlation between the PROs and disease activity due to the relatively low disease activity of our patient cohort. Further, disease activity and physical function are separate constructs, which may not be that well correlated to begin with. Future studies should evaluate the responsiveness of the PROMIS measures relative to disease activity over time and/or use a larger sample size.
In our sample of patients in the rheumatology clinic at SickKids, we found that the PROMIS tools improve in areas where the CHAQ is relatively weak. Specifically, the PROMIS tools exhibited no ceiling effects, had a wider distribution of scores, and were preferred by patients over the CHAQ. Future studies should aim to establish the responsiveness of the PROMIS measures over time in these patient cohorts to further evaluate if they would be appropriate substitutions for the CHAQ.
Footnotes
BMF holds the Ho Family Chair in Autoimmune Diseases. The remaining authors have nothing to disclose.
- Accepted for publication November 10, 2020.
- Copyright © 2021 by the Journal of Rheumatology