Patients and methods

Results

The combined-treatment group included 76 and the sulfasalazine group 79 patients. The groups were balanced in important demographic and prognostic variables.9 ,18 At week 16 the mean improvement based on the pooled index was 1.4 for the combined-treatment group and 0.7 for the sulfasalazine group (p<0.0001). At week 28 these values were 1.5v 0.8 (p<0.0001). In the combined-treatment group rates and rapidity of ACR 20, 50, and 70 improvement were similar to those reported in recent trials on anti-tumour necrosis factor (anti-TNF) treatment (see below).

At week 16, most measures indicated large improvement from baseline in both groups (SRM 0.4–4.1, ES 0.3–3.2), and all measures except patient's global assessment of health (VAS) significantly distinguished between combined treatment and sulfasalazine (table 2, fig 1). Statistics of responsiveness were larger in the combined-treatment group than in the sulfasalazine group, confirming a priori assumptions of greater improvement in this group. The relative responsiveness ranking of measures was similar in both groups, suggesting the ranking is stable over a broad range of relevant response. However, in the sulfasalazine-only group the absolute differences in responsiveness between measures were less, in proportion to the overall decreased response. All indices (that is, pooled index, MACTAR, DAS, and count of improved core set measures) were—in both treatment groups—considerably more responsive than single core set measures such as tender joint count. The only exception to this was the highly responsive single item patient's assessment of change in disease activity on a seven point Likert scale. The responsiveness of most single measures was satisfactory but not equal (for example, high responsiveness for pain and ESR, lower for tender joint count and CRP). A confidence interval smaller than 0.5 around the SRM estimates indicates that a difference between SRMs of 0.35 or greater would be significant when tested at the two sided 0.05 level. The results at week 28 were generally similar (table 3, fig1).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Comparison of responsiveness and discrimination performance of end points. The order of the measures evaluated corresponds with that of table 3. Dark bars: standardised response means of the combination group at 16 (black) and 28 weeks (dark grey). Light bars: standardised response means of the sulfasalazine group at 16 (light grey) and 28 weeks (white). Dots: unpaired t values of the between-group comparison at 16 (black) and 28 weeks (white). pL = points Likert; MACTAR = McMaster Toronto Arthritis patient preference questionnaire; DAS = disease activity score; AIMS = Arthritis Impact Measurement Scale; HAQ = Health Assessment Questionnaire; VAS = visual analogue scale; ESR = erythrocyte sedimentation rate; RS = rating scale; CRP = C reactive protein; SG = standard gamble.

The format of patient assessment of disease activity, physical impairment, and global wellbeing strongly influenced responsiveness (for a description of the formats see ). The item in the MACTAR interview that asked for change of disease activity (seven point Likert scale) proved to be most responsive, patient's global assessment of health indicated on a VAS, least responsive. MACTAR, HAQ, some AIMS subscales, and single item patient global assessment of physical function were not equally responsive. The utility rating scale showed responsiveness close to the patient global assessment of disease activity, whereas utility measured by standard gamble was the least responsive of all measures.

Analyses on floor and ceiling effects showed that, of the core set variables, ESR and tender and swollen joint count were vulnerable to a certain degree of floor effect, with respectively 17, 15, and 15% of the patients in the lowest one sixth segment of the scale. Global and pain assessments, and also the HAQ had fewer patients that scored at the extremes of the scale.

The ranking for between-group discrimination showed interesting trends. This is best seen in fig 1: highest t values (that is, most discriminative power) were found for pooled index, count of core set measures improved by 50%, DAS, HAQ but, also, grip strength. Between the two assessments a catch-up effect is seen in the sulfasalazine group: whereas improvements in the combination group were already maximum at week 16, the sulfasalazine group improved further between week 16 and week 28, resulting in a smaller between-group difference (and thus a smaller t value).

At 16 weeks, ACR 20% improvement and EULAR response criteria showed large χ² values, consistent with significant differences in response between treatment groups (table 4). The discriminatory performance of these criteria ranks high among all the measures tested (table 4): based on p values a χ² value of 8 roughly corresponds with a t value of 3; similarly, a χ² value of 12 corresponds with at value of 4, and a χ² value of 25 with a t value of 5. At week 28 the differences between the groups were smaller. At week 16, modification of the percentage value in the ACR improvement criterion between 0% (no improvement, no worsening) and 50% did not change its discriminatory capacity; at week 28, this was also true for the 70% cut off point (table 4). The ACR and DAS remission criteria did not show a significant between-group difference.

Discussion

This study is the first independent confirmation of the responsiveness of the WHO/ILAR core set measures and response criteria in a trial of patients with early RA. In addition, it lends strong support to the use of other indices in such a trial. The conclusions are strong because they are based on the findings in two groups of patients with a high and moderate level of expected response. They extend the validity of both the core set and the ACRresponse criteria, because these had initially been selected, designed, and tested mainly in placebo controlled studies.

The fact that indices are more responsive than most single measures is not surprising, as combining measures (or items in a questionnaire) reduces scatter. Even a simple count of improved core set outcome variables proved to be a very responsive index, especially at the 20% threshold. Directly asking for change can also reduce scatter, even though the answer may be biased towards the current condition. Evidence for this is shown by the high responsiveness of the patient change question and the MACTAR (that incorporates many change items). The responsiveness of functional scales may be partly explained by the fact that they generally comprise several items in a multi-item questionnaire. Nevertheless, a set of two physical function questions on a six point Likert scale was also responsive.

The strong influence of format and content of the patient's global assessment questions on responsiveness is worrying. Similarly, the responsiveness of pain as a measure depends on the format. It is likely that the focus of doctors (or other assessors) is on the patients' disease, but this seems not always be the case for the patients themselves. Although not specified in great detail in the original formulation of the WHO/ILAR core set, we advocate focusing the format of patient oriented instruments on the disease, and paying close attention to the exact wording of the question(s).

Utility scores are advocated as a generic measure of treatment benefits. The two methods to derive utilities proved to have quite different levels of responsiveness. The rating scale (which is a patient preference rather than a true utility) performed adequately (comparable with observer's global assessment on VAS). However, the standard gamble method (a true utility because choices are made in a situation of uncertainty) showed low responsiveness. Economists prefer standard gamble because it conforms better to theoretical principles, but in practice its application was hindered by limited comprehension of the method by our patients and their risk aversive attitude. This phenomenon has been seen before in patient groups with a non-fatal or chronic disease.31 ,32

The data on between-group discrimination must be interpreted with caution. The extent of differences between the groups was not known before the trial, and might have been large in comparison with expected differences in current and future head-to-head trials. Nevertheless, the results are unique and extremely interesting as they indicate that responsiveness, the ability to detect change, may not parallel the ability to discriminate between different levels of response. Both the ACR improvement criteria (at various percentage levels) and the EULAR response criterion showed excellent discriminatory ability. This is at odds with the other trials in the review of Felsonet al, who concluded that 20% remained the best cut off point for the ACR criteria.33 A possible explanation is the relatively large contrast between treatment groups in the COBRA study.34 Other indices were also better discriminators than most single measures, with the exception of grip strength. In contrast, the discriminatory capacity of the MACTAR, though good, was less than expected based on its excellent responsiveness. This difference in performance between the HAQ and the MACTAR is hard to explain, and will need replication in other studies. Grip strength was included in the design of the trial based on the work of Anderson et al. 22 Despite its good performance in trials up to 1989, grip strength was eventually excluded in the core set for reasons of redundancy. Nevertheless, in early RA the fact that it is a composite measure of hand function with pain, swelling, stiffness, and muscle strength may contribute to its excellent performance. Muscle strength, particularly, may be a physical function variable with potency in early and established RA.35

From published reports we know that different responsiveness statistics—also those that are solely based on change from therapeutic intervention—may36 or may not37 yield different rank orders. In general, rankings based on paired Student'st test values and SRM will only be discrepant when different sample sizes are used for different measures. SRM is least influenced by sample size as it avoids the use of standard error of the mean in the denominator. Sample size was not an issue in this report as few values were missing. ES and SRM generally yield similar ranks, though discrepancies occur when the within-group SD at baseline (the denominator in the ES calculation) differs much from the SD of within-group change (the denominator in the SRM calculation). Obviously, ES cannot be calculated for measures that directly evaluate change, because they do not have a baseline variance. It may be a typical feature of these transitional measures—and indices that include change questions, such as the MACTAR—to pair a large SRM with a relatively small unpaired t value.

Despite their strong evidence, the data represent only one study in one subgroup—that is, early RA. The generalisability of our findings may be slightly limited, as the effects of treatment in the combined-treatment arm were large compared with many other trials in RA, but similar to those seen in recent anti-TNF trials. A meta-analysis of effectiveness of low dose corticosteroids in RA reported somewhat smaller ES in measurements of grip strength, swollen and joint tender count, and ESR (0.4–1.0) than we did; in particular, the ES values found in the corticosteroid treatment arms were smaller.38 However, the effect in the sulfasalazine group resembles that found in trials of methotrexate and intramuscular gold.39-42 Thus for studies of such moderately effective drugs, the ranking of the sulfasalazine group might be more appropriate.

Analyses on floor and ceiling effects showed that ESR and tender and swollen joint count were vulnerable to a certain degree of floor effect. The study's inclusion criteria towards disease duration and disease activity, with evaluation based on ESR and joint counts, probably prevented a serious floor effect in the study group. With global and pain assessments on a visual analogue scale, people tend to put their mark somewhere at the middle of the scale.

Buchbinder et al studied the ability of end points to discriminate between treatment effects in a placebo controlled trial of cyclosporin in RA.43 As the difference between cyclosporin and placebo was the primary study question of that trial, their approach is similar to the post hoc discrimination tests between treatment groups in this report. Compared with the COBRA study, differences between treatment groups were smaller for ESR and swollen joint counts but similar in other measures. They found doctor's and patient's global assessments (measured as a change question), as well as the AIMS pain subscale to be most discriminatory, and ESR and pain (five point scale) to be least discriminatory, with all other core set measures, including another doctor's global question, the HAQ, and a modification of the MACTAR (that is, PET), falling in between. These results agree with our observation on the importance of the exact format of the questions. The discrepancy found in the ESR is expected: lack of responsiveness of ESR is well known during treatment with cyclosporin. More surprising is the relatively poor performance of the physical function questionnaires. It may be that the cyclosporin trial included patients with longstanding disease and more fixed disability that was less likely to respond to treatment. Patients in the COBRA study had a median disease duration of only four months.

Differences in responsiveness, and especially discrimination, have important implications for trial design. The use of responsive and discriminative measures allows reduced patient numbers or detection of smaller—yet relevant—differences between groups. This is important especially in trials of early RA. Simpler trial design through use of a limited number of measures saves costs and effort, and facilitates interpretation of the results. In routine patient care, use of a limited number of highly responsive measures facilitates the collection and interpretation of long term follow up data. Obviously, additional measures should be applied according to the characteristics of the individual patient.

In summary, this study convincingly shows that responsiveness and the ability to discriminate between different levels of response are not identical concepts. The data provide strong evidence for the responsiveness and discriminatory capacity of the WHO/ILAR core set as well as the ACR and EULAR response criteria in the study of moderately and strongly effective drugs in early RA. However, where information is elicited from the patient, researchers should select and focus their instruments on the disease, as performance is strongly dependent on the exact format of questions.