Rheumatoid Arthritis Disease Severity Indices in Administrative Databases: A Systematic Review

MATERIALS AND METHODS

This study was developed according to the Consort Group’s Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, and followed a protocol that prespecified study selection, eligibility criteria, quality assessment, and data abstraction²⁵.

RESULTS

Studies included

The literature search identified a total of 536 studies that matched the predefined search terms. Three were additionally identified by hand-searching reference lists. Of the potentially relevant studies retrieved, the majority were excluded because the data were not from and/or could not be derived from an administrative data source (Figure 1). A total of 2 studies were included in the final analysis and were judged to be of sufficient quality.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

The selection process of studies evaluated for inclusion in the systematic review.

DISCUSSION

Through this systematic review, we found 2 studies that aimed to develop or validate an RA disease severity index for use in administrative database research.

Ting, et al²⁴ built a complex RA severity index, the CIRAS, as a function of a previously developed medical records-based severity index, RARBIS. Although the CIRAS was derived from the selection of the best model predicting the RARBIS, it offered only a moderate correlation with the RARBIS. Their study has some limitations. The investigators used as their reference standard the RARBIS, previously shown to correlate only moderately with disease activity, and for which the reliability has not been thoroughly assessed. Moreover, they used their reference standard to develop the new index. In doing so, and by using the same group of patients in which the new index was developed, some correlation between both indices was expected³². Moreover, variables included in the CIRAS were selected because they were part of the best model identified through diverse selection procedures, not because of an a priori hypothesis that they were related to disease severity, although most variables (extraarticular involvement, RF testing) would be expected to correlate with clinical disease, and so they do provide face validity to the CIRAS. Some have questioned whether components of the RARBIS are appropriate measures of disease severity (e.g., joint surgery) or whether they principally represent aspects of irreversible damage, although joint surgery is obviously likely a correlate of prior disease activity and overall disease severity¹⁴. Although the CIRAS index may allow adjustment of some aspects of RA disease severity, further research is required.

Wolfe, et al³⁰ studied a large RA cohort to determine if treatment variables could predict disease severity³⁰. Their model (including current use of a specific DMARD and/or biologic and lifetime number of these drugs) only partially discriminated patients with high disease severity from those with low severity, as defined by the PAS. Patients with high severity scores were found at both ends of the therapeutic spectrum: those not currently taking a DMARD or a biologic, and those currently taking both a DMARD and a biologic. Again, some limitations might be noted; most importantly, the reference standard was a patient-reported measure, which likely led to significant misclassification of disease severity. It is unclear why the authors decided to exclude patients enrolled in a safety registry. They stated that this was done to prevent selection bias and avoid including participants in a registry who typically have more severe disease. Since patients in the NDB cohort came from the general practice of US rheumatologists, this exclusion criterion may have prevented the investigators from evaluating the full range of disease severity ideal for validation of an index. Further, the study findings may not be generalizable to other countries where a larger proportion of the population is covered by a government medication plan and must satisfy strict clinical criteria (often based on disease severity) to be reimbursed for biologic agents. Thus, it is possible that, in other settings, the use of demographic and treatment variables may better distinguish RA severity.

While performing our systematic search, we found 1 study that assessed the relationship between the number of visits to a rheumatologist and the changes in functional disability (measured by the HAQ) in a community-based cohort of 127 patients with RA³³. The study by Ward, et al was not included in our final set of articles because it did not specifically aim to develop a measure of disease severity and did not use an administrative data source. However, this study is of particular interest since it offers the opportunity to examine if rheumatology visit frequency is a good correlate of disease severity, as measured by the HAQ. Adjusting for potential confounders, the investigators observed a U-shaped relationship between the average visit frequency and the rate of progression of functional disability (the minimum rate of progression corresponded to a visit frequency of 7 visits per year). Although that study was conducted before the biologics era — and at a time when many visits to rheumatologists were for intramuscular gold injections — it emphasizes the complex relationship between medical visits and RA severity. On one hand, infrequent visits may reflect inactive disease, but may also occur among noncompliant patients, who are more likely to have worse disease. Similarly, frequent visits may indicate high adherence to medical management or may represent poorly controlled symptoms that warrant close monitoring. Therefore, more work is needed to assess whether or how visit frequency can independently be used as a correlate of disease severity in health-claims databases. Interestingly, comparison of unadjusted and adjusted risk ratios from observational data do suggest that “proxy” measures of disease severity (like rheumatology visits or orthopedic interventions) may partly correct for biases like channeling³⁴.

An alternative to an RA severity index is the use of propensity scores. This is a method for producing treatment effects adjusted for nonrandom assignment, which is a problem in observational RA studies, since presumably treatment is preferentially used in more severe disease. A propensity score is based on the conditional probability of receiving a particular treatment given certain patient characteristics; patients with the same propensity score have theoretically the same probability of receiving treatment¹⁶. In contrast to other approaches to address confounding by indication, including restriction, stratification, matching, or multivariate adjustment, which are limited by the number of variables that can be controlled for simultaneously, the propensity score method can handle a large number of covariates³⁵. This method creates a single variable (the propensity score) that represents the combined effect of each of the variables considered upon treatment initiation, which can be used for restriction, stratification, matching, or modeling³⁵.

For example, unadjusted results from an observational study of the effect of anti-TNF therapy on malignancy very likely suffer from confounding by indication, which is difficult to control for completely because the number of variables that potentially differ between treated and untreated patients is large. As mentioned, one approach to address confounding by indication is the use of a matched cohort design, in which unexposed subjects with a specific covariate level (or pattern of covariates) are matched to exposed subjects with the same characteristic. However, matching works well with only a few variables and becomes more difficult as the numbers of matched variables increase. Since the reasons for allocation of anti-TNF agent may be complex and based on numerous variables (e.g., not only disease severity but also potentially history, healthcare access, etc.), it is likely that attempts at exact matching on all these variables will fail to find appropriate matches despite large sample sizes obtained from administrative databases (and failure to match on all relevant variables will result in residual confounding). Propensity scores avoid this by combining all the covariates into a single value that can serve as a matching variable, providing excellent balance between treated and untreated groups with respect to the covariates at hand³⁵.

The propensity score model can include multiple variables that are not related to disease severity. Indeed, a propensity score is tailored to the treatment-outcome relationship under study, compared to a “one-size-fits-all” index of severity. It makes use of all available variables present in the database under study that predict treatment exposure (and outcome), as opposed to an index of severity designed for widespread use, which would rely only on variables available in most administrative databases. Moreover, propensity scores developed within an administrative database can provide a relatively detailed picture of a patient’s healthcare use, allowing integration of healthcare use variables such as counts of physician visits, hospital stays, and laboratory tests performed³⁵. As noted, these measures of healthcare use may represent proxies for factors influencing medical decisions, including disease severity.

Although propensity score methods have the advantage of producing “quasi-randomization” in observational studies through the use of rich exposure models, the generalizability of their results is limited and they do not provide adjustment for unobserved confounders¹⁶. The propensity score can control only for known confounders that are measured and available in the dataset in question. Further, certain types of variables entered into the propensity score model can actually introduce more bias, so careful consideration is required during the variable selection process^16,34. Because of these limitations, the use of propensity scores is not a “magic bullet” to completely eliminate confounding by disease severity. Hence, some authors prefer to continue to adjust for factors that are known or believed to be confounders, as opposed to using propensity score adjustment.

Instrumental variable analyses are another tool that could be used to deal with confounding by indication³⁶. This method has the potential to adjust for both measured and unmeasured confounders of the exposure-outcome association under study. The concept is the following: the effect of an exposure on an outcome can be identified by making use of a relationship between the exposure and another variable, the instrumental variable (IV). However, to produce unbiased estimates, strict assumptions have to be fulfilled: (1) the IV must be strongly associated with the exposure; (2) the IV must not have a direct effect on the outcome; and (3) the IV must be independent of the exposure-outcome confounders³⁷. The difficulty with RA disease severity is to find a perfect surrogate variable, satisfying the required assumptions for IV. Indeed, it has been argued that although IV can be useful in case of moderate confounding, they are less useful in the case of strong confounding, which is presumably the case for disease severity, because strong instruments are not always available, and the required assumptions are often violated³⁸. Traditionally, well-recognized IV have been genetic markers, which are obviously not available in administrative databases. Thus, to date this approach has not been used to address confounding by RA disease severity in administrative database research, and one can arguably ask if good candidates for IV actually exist in most administrative databases.

Our systematic review highlights the absence of a comprehensive claims-based index of RA severity, and the paucity of studies aiming to develop one. In administrative database research, it is crucial to adequately measure RA activity/severity to reduce bias from confounding by disease severity. To date, only a few investigators have addressed this challenge, and we applaud their efforts. Further concerted effort from experts in the field is needed to define, develop, and validate a widely applicable measure of RA disease severity for use in administrative databases research. Such a measure could be used by different investigators to perform sensitivity analyses of the influence of confounding by disease severity on their effect estimates, which would enhance the validity and usefulness of the study findings. In the future, results obtained through this approach could eventually be compared with those alternatively obtained with a propensity score model using a minimal set of included covariates. As administrative database research has many advantages, particularly in relatively rare conditions such as RA, it is likely that this area of rheumatic disease research will fuel a lot of interest in the next few years. Until existing methodological gaps are filled, authors can only adjust for disease severity using 1 or more of the available methods, and must always include in their discussion the direction and magnitude of potential effects of residual confounding.