Osteoarthritis (OA) is the most common rheumatic disease, and is becoming a major problem of public health with the ageing of the population in developed countries.¹ OA of the hand is very common. Its prevalence and incidence vary with the definition used. Most studies used the radiographic definition described by Kellgren and Lawrence.² When this case definition was used hand OA was identified in 67% of women and 55% of men in the Rotterdam Study,³ a population-based cohort (age ⩾55 years).

Hand OA could represent a valuable model to evaluate structure-modifying treatments in OA. Various methods have been proposed to assess the radiological severity of hand OA and to score the progression of damage over time.^{2,4,5,6,7,8,9,10,11,12,13,14} Additionally, atlases have been published to help reduce differences in interpretation of the OA grade between observers.^15–17 Indeed, particularly the KL method allows a wide range of interpretations for each grade,^18,19 reducing interobserver reliability. However, the most important issue to consider is the intraobserver reliability, determinant to calculate the smallest detectable difference (SDD)²⁰ and accounting considerably for the sensitivity to change.

The methods proposed differ by (1) the number of hand joints scored; (2) the radiographical features scored: osteophytes, joint space (JS) narrowing, subchondral bone sclerosis, bone cysts, erosion and deformity; (3) the respective importance attributed to each radiological feature in the score; (4) the way of scoring: semiquantitative scoring or global estimate of a qualitative stage of OA (ie, “erosive” or “remodelling” stage in the Verbruggen score^17–19) and (5) the summation. In addition, the way these scales have been published is not standardised and does not permit direct cross-comparisons between them through publications. Information is therefore missing regarding the intrinsic and comparative metrological qualities of these scales.

The aim of our study was to compare in the same sample of patients the precision and sensitivity to change of four radiographic scoring methods proposed to assess the severity and progression of structural changes in hand OA.

PATIENTS AND METHODS

Source of patients

Hand radiographs were obtained from patients included in a randomised, 1-year prospective, multicentre, double-blind, placebo-controlled symptom-modification trial in hand OA. The study had been approved by the ethics review board of the Pitié-Salpêtrière Hospital (Paris, France). Patients were outpatients with symptoms of hand OA (according to the American College of Rheumatology criteria²¹), aged 45–75 years, who gave their written consent to participate in the trial before their enrolment. The level of global pain in the hands during the previous 24 h and functional impairment (using the Functional Index for Hand OA (FIHOA)²²) were assessed at baseline. The minimum symptoms required at entry were a pain level ⩾35 mm on a 100 mm visual analogue scale and an FIHOA score ⩾5 (range 0 to 30).

Selection of radiographs

Pairs of radiographs of 105 patients obtained at baseline and month 12 (M12) were selected at random from among the radiographs of all included patients.

Radiographic techniques

Since the trial from which radiographs were extracted was not a structure-modification trial, no particular specification was given in the protocol as to how to perform hand radiographs. The only recommendation was to obtain an anteroposterior view of both hands on the same film. Radiographs could be either plain or digitalised images and could vary in size. After verification, baseline and M12 radiographs were obtained in the same way for a single patient in 90% of the cases (ie, same magnification). Patient’s radiation exposure was very low (<0.01 milliSievert (equivalent to <0.5 chest x ray).

Blinding process for radiographs

Radiographs were identified by the randomisation number allocated to the patient at study entry. A letter (A or B) was randomly assigned to code the time sequence (baseline or M12) on radiographs. Therefore, each radiograph was identified by a letter and a number. The two coded films available for each patient were gathered in an envelope.

Reading procedures

Two trained readers (CC, EM) performed the scorings separately. They were unaware of the patient’s identity, drug assignment, time sequence of the radiographs and each other’s measurement. Before starting the readings, they underwent two training sessions together on the same radiographs to identify discrepancies and inconsistencies with respect to each scoring method and to try to reduce their disagreement.

Each radiograph was scored according to four methods (see below). Radiographs were read on a horizontally positioned light box to facilitate the scoring of each hand joint. In some cases, to help to assess specific difficult radiographic features, it was agreed between the two readers to use a magnifying glass. Radiographs were read in pairs: the two views corresponding to each patient were read at the same time. Radiographs were read in sessions of 30 radiographs (representing 15 pairs, during each session). Therefore, seven sessions were used for the first reading. If necessary, a break was planned during each session so as not to exceed more than two consecutive hours of reading. For the assessment of cross-sectional and longitudinal intraobserver reliability, 60 pairs of radiographs (baseline–M12) were selected at random from among the 105 pairs read initially. These sets of radiographs were read a second time, with a minimum time interval of 30 days between first and second readings. Altogether, 330 radiographs were scored using the four scoring methods by each of the two readers. Readings were performed over a 4-month period.

Radiological scoring methods

The four methods compared are listed in table 1, with the numbers and sites of hand joints examined for each. Radiographs were scored according to:

1. A global scoring, during which the reader had to decide whether the joint assessed was osteoarthritic or not (yes/no).⁹ A total of 32 joints were assessed (range 0–32).

2. KL grading,² as specified by the authors in their atlas.¹⁵ We took into account the modifications suggested by Kellgren-Lawrence themselves, reproduced by Lane¹³ and discussed by Spector.^18,19 Briefly, each of the 30 joints examined was scored from 0 to 4, according to the presence and size of osteophyte(s) and according to JS narrowing: 0, no OA; 1, doubtful OA; 2, definite minimal OA; 3, moderate OA and 4, severe OA. The score ranges from 0 to 120.

3. Kallman radiographic scale (initially validated by scoring left hands in a cohort of men),^5,6 where 24 joints (all, but the metacarpophalangeal joints) are scored for six radiographical features according to a seminumerical scale: osteophytes (0–3), JS narrowing (0–3), subchondral bone sclerosis (0–1), subchondral bone cysts (0–1), lateral bony deviation (⩾15°; 0–1) and bone erosion (0–1). The score ranges from 0 to 208.

4. Verbruggen numerical scoring systems for the anatomical evolution of OA of the finger joints.^10–12 We used the anatomical grading scale proposed by Verbruggen, the only one allowing cross-sectional comparison with other semiquantitative radiographic grading systems. The score ranges from 0 to 218.4. Of note, thumb base joints (scapho-trapezial and trapezio-metacarpal) are not considered in this score (table 1).

Statistical analysis

The following data were recorded at baseline for the 105 patients from whom radiographs had been selected: age, gender, weight, height, mean disease duration, handedness, patterns of joint involvement with respect to the most painful joint at entry, pain level on a visual analogue scale, FIHOA score²² and radiographic scores.

The ease of use was assessed by the mean time (min) taken to perform each scoring method.

To compare reliabilities and responsiveness of the scores, as they have very different ranges (and since it is known that ICCs increase with the variability of a sample), we decided to normalise the individual scores by the maximal value obtained during the readings. The maximal values recorded were 28 for the global score, 75 for the KL, 123 for the Kallman and 108.7 for the Verbruggen scores. Data were normalised by these values, then multiplied by 100 to obtain scores ranging from 0 to 100.

Reproducibility of the four radiological scoring methods evaluated by intraobserver and interobserver reproducibilities was assessed using the ICC²³ and the Bland–Altman plotting method,²⁴ which allows the calculation of the SDD²⁰ (ie, the amount of detectable change above the random measurement error). SDDs were calculated for both interobserver and intraobserver reliability according to a previously described method.²⁰ Estimates of ICC were derived in the framework of a two-way fixed effect model. 95% CIs were estimated by the method described by Shrout and Fleiss.²⁵ Differences between readers (interobserver reliability) and between readings (intraobserver reliability) were calculated using the Bland and Altman approach.

Sensitivity to change of each radiographic scoring method was estimated on the basis of differences in each score between M12 and M0 using the standardised response mean (SRM  =  mean change /SD of change). The 95% CIs of SRM estimates were calculated using the jackknife technique.²⁶ All statistical analyses were performed using software SAS V.6.12.

RESULTS

Table 2 shows the descriptive clinical data of the 105 patients whose radiographs were selected. Table 1 provides the general results of the readings for each scoring method and for each observer.

Ease of use

The mean time used to perform each scoring method is also given in table 1. The Kallman method was the most time-consuming, while the global and Verbruggen scorings were the most rapid to perform.

Interobserver reproducibility

ICC values and their 95% CI on baseline scorings and on the scorings of changes between baseline and M12, mean differences on baseline scorings (SD) and SDD are provided for each method in table 3.

ICC values were 0.859 for global score, 0.951 for KL, 0.706 for Kallman and 0.996 for Verbruggen, exhibiting a good inter-observer reproducibility for all scores. However, Verbruggen and Kellgren scores rated the best.

The Bland and Altman plotting method results for interobserver reproducibility of scorings at baseline and measurements of change between baseline and M12 by reader 2 appear in figs 1 and 2, respectively, for the four scoring methods.

Intraobserver reproducibility

Cross-sectional intraobserver reproducibility of scorings performed on baseline radiographs

ICC, mean (SD) differences between repeated measurements of baseline radiographs and SDD are indicated in table 3 for each reader and each grading scale. ICC values (95% CI) were very high for both readers, whatever the scoring method used, ranging from 0.922 to 0.999. The lowest ICC was obtained by both readers with the global assessment.

The Bland and Altman plotting method results for intraobserver reproducibility of scorings at baseline by reader 2 appear in fig 1 for the four scoring methods.

Longitudinal intraobserver reproducibility of scorings on baseline and M12 radiographs

ICC, mean (SD) differences between repeated scorings of changes between M12 and baseline radiographs, and SDD are also given in table 3 for each reader and each grading scale.

The Bland and Altman plotting method results for intra-observer reproducibility of the scorings of changes between M12 and baseline by reader 2 appear in fig 2 for the four scoring methods.

ICC values (95% CI) were also very high for each observer and all four methods, as shown in table 3, ranging from 0.939 to 0.998. Again, the lowest rate was obtained for the global assessment, and the highest for the KL scoring method.

Both readers showed very good precision, as assessed by the ICC. Reader 2 was more reliable for Verbruggen and global scoring, as assessed by both ICC and the Bland–Altman graphics, whereas reader 1 was more reliable for Kallman scoring (data not shown). Both exhibited the same high level of precision with respect to the KL method.

Sensitivity to change over time

The SRM values were moderate, ranging from 0.24 (KL scoring method—reader 2) to 0.29 (Kallman scoring method—reader 2), as shown in table 4. The estimate of the precision of the SRM calculated was performed using the jackknife technique: 95% CI jackknife SRM are given in table 4. According to the values calculated in this study, the four studied scoring methods could be considered sensitive over a period of time as short as 1 year. Reader 2 was a little more sensitive to change than reader 1 for Kellgren, Verbruggen and global scoring methods (data not shown for reader 1). Kallman seemed a little more responsive for both readers (data not shown for reader 1).

DISCUSSION

We report the results of the first study performed to compare four radiological scoring methods in hand OA. All four scales performed well with respect to inter-, intraobserver reliability and sensitivity to change over a short period of time (1 year). The Verbruggen and Kallman scales performed better with respect to reliability. The Kallman method was slightly more sensitive to change.

Hand OA could be an interesting model to study structure-modifying therapeutics. In addition, assessing hand OA could be of interest to predict further incident hip or knee OA, as shown recently.²⁷ However, these issues require a valid and reliable tool to assess the severity and progression over time of hand OA on radiographs.

None of the proposed radiological scales,^{4,5,6,7,8,9,10,11,12,13,14} have, as yet, proved to be better than another: there has been no direct cross-comparison to assess their reliability and responsiveness.²⁸

This study allowed the evaluation of the metrological properties (ie, simplicity of use, validity, reliability, sensitivity to change and discriminant capacity) of the four scales compared: global assessment, KL, Kallman and Verbruggen. It was carried out on a large sample of radiographs from 105 patients, representative of all stages of the disease. Only two studies previously examined the precision of radiological scoring methods in hand OA: a semiquantitative global assessment (four-grade scale) was compared with a dichotomous global scale (OA: yes/no) by Dougados.⁹ The global assessment exhibited a higher intraobserver precision than the four-grade scale. Hart et al²⁹ compared KL grading with an assessment of JS and osteophyte on a four-point scale, and found that all three methods performed well with respect to both inter-reliability and intra-reliability, the latter rating being better.

The quality of the radiographs did not seem to interfere greatly with the scoring, with the exception of subchondral sclerosis assessment, which requires well-defined radiographs—that is, not too overexposed. Scoring digitised or plain films did not seem to modify the quality of assessment, as reported previously.^30,31

The global and the Verbruggen scoring methods were less time-consuming (1.5–1.6 min to perform). Cross-sectional and longitudinal interobserver reproducibilities were good for all methods, using ICCs. Both readers exhibited a good intraobserver precision, on baseline radiographs and on changes between baseline and year one. Kallman and Verbruggen scorings were slightly better for the cross-sectional study and assessment of changes (responsiveness). The Kallman method might have shown a better responsiveness, because it assesses six radiological signs. However, it may perform better without some assessments likely to create a background noise (cysts and subchondral sclerosis, for instance). The Verbruggen score, despite it being a global scale, performed well with respect to reliability, but was a little less responsive than the Kallman method. This could be explained by the weight of the “remodelling” phase, which is a final phase unlikely to change over time. For cross-sectional epidemiological studies of the prevalence of hand OA in a population, KL and Verbruggen methods performed better than the global and Kallman methods.

We used the scores as published by their authors, but some modifications could help to increase their precision. Additionally, these methods should be compared for the same patterns of digital joints. How the scales perform when only distal interphalangeal and proximal interphalangeal joints are scored (excluding thumb base joints and metacarpophalangeal joints) remains unknown, as is unknown how the Kallman method performs if cysts and subchondral sclerosis assessments are deleted. The performances of the Kallman and Verbruggen scores modified in this way should be explored through further studies. Furthermore, it must be noted that according to the progression of hand OA described by Verbruggen, the remodelling (R) phase represents the most severe and final stage, whereas if scored according to Kallman, the same joint might exhibit an improvement of OA with a JS widening.

Using the Bland and Altman approach allowed the calculation of the SDD, which is based on the measurement error and corresponds to the minimal amount of progression that can be reliably distinguished from random error measurement. It allows calculation of cut-off points for the definition of progression. Our study provides data to calculate such cut-off points that could be used in future structure-modification trials.

There are some limitations to this study: we did not assess discrepancies to identify how each radiological feature accounted for these discrepancies (ie, osteophyte, JS narrowing, subchondral sclerosis or cysts, erosion or lateral deviation assessment). Despite two preliminary training and discussion sessions regarding the scoring of each radiological sign, it was not possible to completely reduce the interpretation of the observer. For instance, what is a “definite” or a “moderate” osteophyte, or a “possible narrowing of joint space at one point”, as indicated by Kellgren and Lawrence in their description?^2,15 This has already been discussed by Spector.¹⁸ Using published atlases^15–17 did not help to adjudicate discrepancies during preliminary training. Therefore, we decided not to use atlases. Since these methods do not include any quantification of the radiological signs examined, each observer had to set his own rules to obtain a score. The time interval of 1 year between the two radiographs for each patient was short. However, our study showed that trained readers identify radiological changes over such a short period of time in hand OA. Structure-modification trials are usually conducted over longer periods of time (2 or 3 years). It is likely that over such periods, the responsiveness of the four methods (SRM) will be better than that reported in our study.

In summary, all methods exhibited high intraobserver reliability. Although our results seem to favour Kallman (reliability, responsiveness) and Verbruggen (reliability, simplicity to use) scales, the magnitude of the differences between the scales did not strongly discriminate between the methods. The SRMs calculated in our study were very close for all four methods. Studies over a longer period of time, using either original or modified Kallman, Kellgren or Verbruggen scales, are needed to assess further which method should be considered the most reliable and responsive, and therefore be recommended to assess hand OA progression in structure-modification studies or trials.