Abstract
Objective. To compare the performance of an extended model and a simplified prognostic model for joint damage in rheumatoid arthritis (RA) based on 3 baseline risk factors: anticyclic citrullinated peptide antibodies (anti-CCP), erosions, and acute-phase reaction.
Methods. Data were used from the Nijmegen early RA cohort. An extended model and a simplified baseline prediction model were developed to predict joint damage progression between 0 and 3 years. Joint damage progression was assessed using the Ratingen score. In the extended model, prediction factors were positivity for anti-CCP and/or rheumatoid factor, the level of erythrocyte sedimentation rate, and the quantity of erosions. The prediction score was calculated as the sum of the regression coefficients. In the simplified model, the prediction factors were dichotomized and the number of risk factors was counted. Performances of both models were compared using discrimination and calibration. The models were internally validated using bootstrapping.
Results. The extended model resulted in a prediction score between 0 and 5.6 with an area under the receiver-operation characteristic (ROC) curve of 0.77 (95% CI 0.72–0.81). The simplified model resulted in a prediction score between 0 and 3. This model had an area under the ROC curve of 0.75 (95% CI 0.70–0.80). In internal validation, the 2 models showed reasonably well the agreement between observed and predicted probabilities for joint damage progression (Hosmer-Lemeshow test p > 0.05 and calibration slope near 1.0).
Conclusion. A simple prediction model for joint damage progression in early RA, by only counting the number of risk factors, has adequate performance. This facilitates the translation of the theoretical prognostic models to daily clinical practice.
- RHEUMATOID ARTHRITIS
- JOINT EROSIONS
- ACUTE-PHASE REACTION
- ANTICYCLIC CITRULLINATED ANTIBODIES
- RISK FACTORS
Rheumatoid arthritis (RA) is a multifactorial disease with joint damage as one of its hallmarks. An important aim in the treatment of RA, therefore, is the prevention of joint damage, next to relieving symptoms and prevention of disability. The risk for joint damage is not the same for all patients with RA, and in current treatment guidelines, it is advised to take the risk for this unfavorable prognosis into account1,2. Three prognostic factors that are regarded as most strongly determining the probability of joint damage progression in early RA are the presence of anticyclic citrullinated peptide antibodies (anti-CCP) and/or rheumatoid factor (RF), presence of erosions, and high levels of acute-phase response. In cases of persistent high or moderate disease activity because of methotrexate (MTX) failure, patients with 1 of these 3 main factors for unfavorable prognosis at diagnosis are advised to switch to biologics earlier, compared with patients who do not have these risk factors1,2. Moreover, making use of the window of opportunity in the treatment of RA, it is suggested that patients with a high risk for joint damage progression should, at the moment of RA diagnosis, be treated with more intensive therapy, such as MTX with a high dose of prednisone, combination therapy with 2 or 3 disease-modifying antirheumatic drugs (DMARD), or even biologics3,4.
To guide this treatment strategy for an individual patient, several multivariable prognostic models have been developed to predict the risk for joint damage progression in early RA5,6. Interestingly, the models agree on the factors that are important for the prognosis: anti-CCP, high acute-phase reactant, and presence of erosions. However, the importance of the 3 risk factors and their weight in the prediction models vary considerably among the models. To date, none of the prediction models appear to be widely used in daily clinical practice for estimating the risk for joint damage progression and for personalizing treatments. This may be because the translation is not often made from the regression coefficients or OR in the prediction models to a clinically useful risk estimation for the individual patient. In the last few years, matrices have been developed for use in daily clinical practice7,8,9,10. However, in external validation, the discriminative ability of these models appears to be disappointing11,12. We hypothesized that further simplification of the prediction models could increase generalizability and could facilitate the translation from the theoretical prediction model for use in daily clinical practice. For this reason, we aimed to develop a simplified model for the prediction of the risk for joint damage progression in early RA. Simplification is only useful if it does not result in a significant loss of predictive performance. Therefore, the objective of our study was to compare the performance of an extended model and a simplified baseline prognostic model for joint damage in RA based on anti-CCP, erosions, and acute-phase reaction.
MATERIALS AND METHODS
Design
Data were used of the first 3 years of disease from patients of the Nijmegen early RA inception cohort in the Netherlands, who were included from 1985 to 200813. All patients provided written informed consent. In the Netherlands, no formal approval of a medical ethical review board is necessary for this type of observational study.
Patients
Patients with RA were consecutively included in the cohort if they fulfilled the 1987 American College of Rheumatology criteria for RA, had a disease duration of less than 1 year, had no prior DMARD use, and were aged 18 years or older. There were no exclusion criteria13.
Cohort patients were included for the current analyses if radiographs were available at inclusion and after 2 or 3 years of followup. Patients treated with biologic DMARD within the first 3 years of disease were excluded from the analyses because this type of medication could modify the relation between inflammation and joint damage progression14.
Assessments
At diagnosis, clinical data were collected, including presence of anti-CCP, RF, shared epitope (SE), and smoking status (ever or never smoker). The level of inflammation at baseline was assessed using erythrocyte sedimentation rate (ESR), C-reactive protein (CRP), swollen joint count of 44 (SJC) and 28 joints (SJC28), tender joint count of 53 (TJC) and 28 joints (TJC28), general health (GH) on a visual analog scale (VAS), and pain on a VAS.
Radiographs of hands and feet were taken at baseline and at followup after 1, 2, and 3 years. Radiographs at baseline and at 3-year followup were scored according to the Ratingen score15. This was a modification of the Larsen score, evaluating the percentage of joint surface destruction of 38 joints of the hands and feet, graded from 0 to 5 (range 0–190). Joint damage progressed most in the first 3 years of the disease16. Therefore, the change in Ratingen score between baseline and 3 years was the outcome of the prediction model. A subset of radiographs had been read in 2002 by 4 raters (ICC 0.85). For our study, radiographs from 2002 until 2011 were read by 2 raters (ICC 0.95). Joint damage progression was defined as a difference of ≥ 5 Ratingen points between baseline and 36-month followup, corresponding with the smallest detectable change in this cohort, as previously calculated17. Sensitivity analyses were done with ≥ 1 and ≥ 10 Ratingen points as the cutpoint for joint damage progression.
Analyses
First, it was checked whether the main baseline prognostic factors for joint damage progression in RA as mentioned in literature — anti-CCP, ESR, and erosions — were also the main prognostic factors for joint damage progression in our cohort. The association with joint damage progression in the first 3 years of the disease was analyzed using univariable and multivariable logistic regression with the potential predictors: smoking status (ever smoked/never smoked), anti-CCP and RF combined (0, 1, or 2 positive), SE (positive/negative), SJC28 (0–5, 6–10, 11–15, > 15), TJC28 (0–5, 6–10, 11–15, > 15), DAS28 (≤ 3.2, 3.3–5.1, ≥ 5.2), CRP (≤ 5, 6–10, ≥ 11), ESR (< 25, 25–50, > 50 mm/h), VAS pain (0–29, 30–60, 61–100), GH (0–29, 30–60, 61–100), and erosions at baseline (0, 1–5, 6–10, > 10 Ratingen points). In the multivariable model using backward selection with p < 0.20 as the selection criterion, it appeared that anti-CCP, ESR, and erosions were strongly and significantly associated with joint damage progression. SJC28 was significantly but weakly associated with joint damage progression. Therefore, further analyses were done with the 3 main risk factors as known from literature and without SJC28. Age (< 45, 45–60, > 60 yrs at diagnosis) and sex were added to the model to be able to apply the model regardless of sex and age.
Subsequently, 2 baseline prediction models were made, including anti-CCP, ESR, and erosions as the prognostic factors, and also including age at diagnosis and sex. The first model was an extended model, in which prediction factors at baseline were categorized: positivity for anti-CCP and/or RF [positive for 0, 1, or 2, with anti-CCP > 25 U/ml regarded as positive, anti-CCP > 10 U/l regarded as positive in the subset (37%) of posthoc frozen samples, and > 10 U/ml as the cutpoint for RF]; the level of ESR (< 25, 26–50, > 50 mm/h); and the number of erosions (0, 1–5, 6–10, > 10 Ratingen points). Anti-CCP was measured using ELISA immunoscan RA Mark 2 (Euro Diagnostica) and the posthoc frozen samples were measured using EliA-CCP (Thermo Scientific). The prediction score for joint damage progression was defined as the sum of the regression coefficients of the 3 predictors in the multivariable regression model18. The second model was a simplified model, in which the 3 prediction factors were dichotomized: anti-CCP > 25 U/ml (> 10 U/l in the posthoc frozen samples), ESR > 25 mm/h, and Ratingen score at baseline ≥ 1. The prediction score for joint damage progression was defined as the number of prognostic factors present (0–3).
Missing values analysis was performed by evaluating the frequencies and patterns of missing variables using SPSS (IBM SPSS) missing value analysis. Missing values showed no recognizable pattern and were considered missing at random. Missing radiographs at the 3-year followup were imputed using the radiographs at 2-year followup according to the last observation carried forward principle. Missing prediction variables were imputed using multiple imputations with 5 imputed datasets based on the continuous baseline variables ESR, CRP, SJC28, TJC28, VAS pain, GH, age, Ratingen score and the categorical variables smoking status, sex, RF, anti-CCP, shared epitope, and erosions at 3-year followup.
Internal validation was performed using a bootstrapping procedure with 300 repetitions, as usual, for each of the 5 imputed datasets separately19,20. Performances of the extended and simplified models were assessed by calculating discrimination and calibration. The discriminative ability was calculated as the area under the receiver-operation characteristic (ROC) curve. Goodness-of-fit was tested using the Hosmer-Lemeshow test in which a p value < 0.05 indicated lack of fit, and using the calibration slope. A calibration slope of 1.0 meant (in the bootstrap samples) that the prediction rule agreed well with the observations. Results for discrimination and calibration of the 5 imputed datasets were combined according to Rubin’s rules or given as ranges as appropriate20,21.
SPSS version 20.0 and SAS version 9.2 (SAS Institute Inc.) were used for the analyses.
RESULTS
Patients
Until December 2008, 607 patients were included in the Nijmegen early RA cohort at the Radboud University Medical Centre. Seventy-seven patients (13%) were lost to followup during the first 3 years of disease (2% moved, 2% died, 1% stopped because of comorbidity, 5% did not want to participate anymore, 2% had no RA after all, and 1% were lost for an unknown reason). For 90 patients (15%), no radiographs of the hand and feet were available at 2-year or 3-year followups. Fifteen patients were excluded because they had been treated with biologic DMARD during the first 3 years of disease. Anti-CCP was missing in 54 patients (13%) and ESR at baseline in 19 patients (4%) of the study population. Missing baseline variables were overcome by multiple imputation and therefore, data of 425 patients (70%) could be used for the analyses. Radiograph scores at the 3-year followup were imputed in 12% of patients.
Baseline
In Table 1, baseline data of the patients with and without joint damage are presented. Of the 425 patients, 175 (41%) had progression of joint damage between 0 and 36 months, and 250 had no progression. The patients in the group with joint damage progression had a mean age of 54 years (SD 14) and 59% were women. Of these patients, 82% were positive for anti-CCP, 62% had erosions at baseline, and the median ESR at baseline was 40 mm/h. The patients in the group without joint damage had a mean age of 57 years (SD 14) and 66% were women. Of these patients, 53% were anti-CCP positive, 32% had erosions at baseline, and the median ESR was 21 mm/h. Anti-CCP positivity, RF positivity, baseline inflammation levels, and presence of erosive lesions on radiographs at baseline were significantly higher in the group of patients with joint damage progression. These patients were also significantly younger than the patients without joint damage.
Of the patients with joint damage progression, 75% were treated with DMARD monotherapy and 21% with DMARD combination therapy between 0 and 36 months followup. Oral prednisone was used by 25%. Of the patients without joint damage progression, 72% and 20% were treated with monotherapy and combination DMARD therapy, respectively, and 15% of patients were treated with oral prednisone. Use of oral prednisone was significantly lower in the group without joint damage progression (p = 0.013). Other differences in treatment between the 2 groups were not significant.
Extended prediction model: categories of the 3 risk factors
All 3 prognostic variables, anti-CCP/RF, ESR, and presence of erosions, were univariably and multivariably significantly associated with joint damage progression (Table 2). The sum of the regression coefficients resulted in a prediction score with a range between 0 and 5.6. The median prediction score for the patients with joint damage progression was 3.0 [interquartile range (IQR) 2.1–3.7] as compared with 1.3 (IQR 1.0–2.3) in patients without joint damage progression (p < 0.001). An anti-CCP and RF-positive patient with low ESR and no erosions at baseline would have had a prediction score of 1.34, corresponding with a predicted probability of 0.25. An anti-CCP and RF-positive patient with high ESR and no erosions at baseline would have had a prediction score of 2.95, corresponding with a predicted probability of 0.62 (Figure 1A). Usually, an (online) calculator was used for this purpose in the clinic. The prediction model had an area under the ROC curve of 0.77 (95% CI 0.72–0.81; Figure 2). Sensitivity analyses with 1 and 10 as cutpoints for joint damage progression showed a similar area under the curve.
Internal validation was applied using bootstrap samples. The shrinkage factor was 0.90 (95% CI 0.88–0.92). The discrimination according to the area under the ROC curve changed little and ranged from 0.76–0.77. Agreement between predicted and observed probabilities (calibration), as reflected by the Hosmer-Lemeshow test, was adequate in all 5 imputed datasets, with p values ranging between 0.41 and 0.85. The calibration slope was 1.0 (95% CI 0.82–1.21).
Simplified prediction model: dichotomization of the 3 risk factors
In the simplified prediction model, anti-CCP positivity, high ESR, and erosions at baseline were univariably and multivariably significantly associated with joint damage progression. The prediction score of the simplified model represented the number of unfavorable prognostic factors present in a patient (0–3; Table 3). Two percent of patients with joint damage progression had 0 risk factors, 19% had 1 risk factor, 40% had 2, and 39% had 3 risk factors. In the group of patients without joint damage progression, 21%, 38%, 31%, and 10% of patients had 0, 1, 2, and 3 risk factors, respectively (p < 0.001). The number of risk factors present, 0, 1, 2, and 3, corresponded with a probability for joint damage progression of 0.09, 0.25, 0.49, and 0.57, respectively (Figure 1B). The discriminative ability of the simplified model, as calculated by the area under the ROC curve of 0.75 (95% CI 0.70–0.80), was moderately good (Figure 2). Sensitivity analyses with the cutpoints 1 and 10 resulted in the same area under the curve.
In the internal validation procedure, the shrinkage factor was 0.98 (95% CI 0.96–1.0). Discrimination according to the area under the ROC curve ranged from 0.75 to 0.76. Calibration was adequate with p values of the Hosmer-Lemeshow test varying between 0.87 and 0.99. The calibration slope was 1.1 (95% CI 0.89–1.33).
DISCUSSION
The objective of our study was to compare the performance of an extended model and a simplified prediction model for joint damage progression based on the baseline prediction factors: anti-CCP, high ESR, and erosions. The extended model resulted in a prediction score varying between 0 and 5.6, with adequate discrimination and calibration. The simplified model resulted in a prediction score from 0 to 3, also with adequate discrimination and calibration. The discriminative ability of the simplified model was comparable to the extended model. In internal validation, it was seen that the predictions of the simplified model were especially accurate: there was virtually no overoptimism (shrinkage factor 0.98) and the discrimination remained unchanged (area under the ROC curve 0.75). It is generally known that having more risk factors is associated with higher risk of progression. The value of this type of model is that the exact probability can be made available.
Identification of the main prognostic factors for joint damage progression in RA has been done in several studies. Translation of the prognostic factors into a clinically useful prediction model has also been done in other studies by quantifying the risk for joint damage progression for different risk groups. The model of Visser, et al resulted in 9 groups by categorization of CRP, and the number of erosions and positivity for anti-CCP and/or RF7. Fautrel, et al divided the study population into 36 groups by categorization of CRP, erosions, anti-CCP, and SJC288. The simplification of our model resulted in the quantification of the risk for the 4 risk groups (0, 1, 2, and 3). Performances of the models by Visser, et al and Fautrel, et al in our data were in the range of the models developed in our study, with discriminative abilities in the range from 0.76 to 0.79. Simplification of a prediction model might be paid for with loss of performance regarding discriminative and predictive ability. This was also true for our simplified prediction model, although not statistically significant. When the performances of the models are similar, a simple model is easier to use in the clinic. Whether these values are acceptable depends on the consequences of the intended therapy. Misclassification is more worrisome if a therapy is expensive, burdensome, or highly toxic.
The starting point of our analyses was how to use the prognostic factors (anti-CCP, ESR, and erosions) in a simplified baseline prediction model that could facilitate the translation of the theoretical models for use in daily practice. Univariable and multivariable logistic regression analyses confirmed these 3 factors as the most important prognostic factors for joint damage progression. Because the regression coefficients for these 3 variables were about equal, equal importance of the predictors is suggested. SJC28 was significantly but weakly associated in the multivariable model; therefore, we decided not to use this predictor in further development of the simple model. Because age and sex were added as confounders to both models, the prediction scores are independent of age and sex.
In textbooks and guidelines, it is suggested that patients with RA with an unfavorable prognosis regarding joint damage progression should be treated differently from patients with a better prognosis1,2. Commonly, anti-CCP, RF, erosions, and a high level of acute-phase reactant are regarded as factors pointing to an unfavorable prognosis. At the same time, the exact risk for joint damage progression remains unclear. The results of our study showed how the 3 main baseline predictive factors can be translated into a simple risk model for joint damage progression in early RA without loss of predictive performance.
A prediction score extracted from the extended model can guide treatment choices in daily practice by setting cutpoints on the continuous scale of the prediction score. In general, 3 categories can be extracted from the S curve of a prediction model: a safe range at low scores in which patients can be treated with less intensive treatment; an unsafe range at high scores requiring an intensive treatment strategy; and an intermediate category in which additional prognostic tests are valuable or close followup is needed so that treatment can be intensified quickly if remission or low disease activity are not reached quickly. Easier than scoring the prognostic factors is to count them. According to the results of our study, this could result in 4 risk groups. The next step for translation into clinical practice is to study the optimal treatment strategy for each of these 4 categories. This is a step toward personalized medicine in RA.
Next to baseline factors, the level of disease activity over time is also important for the estimation of the risk for joint damage progression. A direct consequence could be to define different treatment targets for disease activity for each of the 4 risk categories22.
Regularly, new possible prediction factors for joint damage in RA are identified, such as the Power Doppler of an index joint or anemia23,24,25. Future research could focus on the additive value of these newly identified prediction factors in a multivariable regression model with anti-CCP, erosions, and ESR at baseline. However, before further development of the simplified baseline prediction model, we recommend that this model is externally validated in other early RA samples to verify the equal importance of the 3 main risk factors for the prediction of joint damage progression.
Deliberately, an old RA cohort was used for the development of the models, with an inclusion period from 1985 until 2008. This might be seen as a limitation of our study, but that is not the case, per se. Patients in our cohort had more joint damage than can be expected in a more recent cohort because in recent years, patients are more likely to experience earlier diagnosis, more intensive treatment, and a milder disease course26. Ideally, prognostic models are developed in patients who were untreated, i.e., following the natural course of disease. The advantage of an old cohort that is less intensively treated is that it resembles the natural course of RA as closely as possible.
The Ratingen score was used for evaluation of joint damage, which is a variation on the Larsen score. Differences between the Ratingen score and the more often used modified Sharp score (mSharp) are that in the Ratingen score, the percentage of damaged joint surface is scored rather than the number of erosions, and joint space narrowing is not scored. However, in the Ratingen score and mSharp score, the same joints are evaluated and the 2 scores are closely correlated27,28.
A simplified prediction model for joint damage progression in early RA, only counting the number of risk factors — anti-CCP, high ESR, and erosions — has adequate discriminative ability and has similar predictive performance as compared with an extended model based on the same variables. A simplified model facilitates the translation from a theoretical model to daily clinical practice and further differentiates in the current dichotomy in treatment guidelines, being at risk or not being at risk for unfavorable prognosis. This can guide treatment choices and is a step toward personalized medicine in RA.
Acknowledgment
We thank all research nurses and rheumatologists for their participation and contribution in data collection, and Thea van Gaalen, Lia Schalkwijk, Carien Versteegden, Erik Brummelkamp, and Marjon de Lange-Brand for data processing.
- Accepted for publication July 14, 2014.