Systemic sclerosis (SSc) is characterised by tightening and thickening of the skin, and involvement of internal organs. Particularly early in the disease, more extensive skin involvement coincides with more severe internal organ manifestation(s), poor prognosis^1,2 and increased disability.^3,4 Derived from the original Rodnan skin score,⁵ several modified and simplified skin score measuring methods have been developed.^2,6,7 The modified Rodnan skin score (MRSS) uses clinical palpation to estimate the skin thickness. At present, the MRSS is considered the most appropriate and reproducible technique for measuring skin involvement in SSc.^2,6,8,9,10 The method is easily used and fully validated.^8,9

The aims of this study were (1) to establish a method for teaching larger numbers of rheumatologists to appropriately perform the MRSS and (2) to evaluate the interobserver and intraobserver variabilities of skin scoring in a group of mixed patients with SSc with both limited and diffuse disease examined by different rheumatologists from centres all over Europe.

METHODS

The fully validated^8,9 modified version of the Rodnan skin thickness score^5,9 was used. A total of 17 skin sites were evaluated, including the face, upper arms, forearms, dorsum of the hands, fingers, chest, abdomen, thighs, forearms and feet. The maximum total score is 51 (MRSS-51) and the grading is 0, normal skin; 1, thickened skin; 2, thickened and unable to pinch; and 3, thickened and unable to move.

Investigators

Teaching course for teachers

The 1st course (Berlin, June 2004), 2nd course (Budapest, January 2005) and 3rd course (Vienna, June 2005) were organised in a similar fashion. First, the “master teacher” (DEF) explained and demonstrated the technique of MRSS to the nine examiners, and examined each patient to establish the standard. Then, the examiners performed the investigations on the patients with SSc. The proportion of patients with diffuse cutaneous SSc was 66–100%, with a range of 33–80% of early cases (disease duration <3 years).

Teaching of a large number of young rheumatologists

A total of 90 young rheumatologists participated from all over Europe at the training in Budapest (January 2005), guided by the same master trainer (DEF). The master trainer gave a further demonstration of the technique to the nine tutors, followed by a discussion of details. Next, with the assistance of a patient with scleroderma, the master trainer demonstrated the details of the technique to the 87 “trainee” rheumatologists. Meanwhile, all tutors evaluated their patients, and after that, reasons for the recorded variability were once more discussed with the master trainer. Thereafter, the 87 trainee rheumatologists were divided into nine groups, each consisting of 5–6 students. Each trainee group had two patients for practice. Following a discussion with the tutor, each trainee investigated another two patients.

Local courses to investigate the intraobserver variability

The first local teaching course (Pécs, April 2005) was organized as described above. Nine rheumatologists investigated 6 patients each a local teacher from the previous course. During the second local course (Pécs, June 2005) 5 examiners who had participated in the previous course in April, each investigated 7 SSc patients.

Statistical calculations

All calculations were made with SPSS/PC+ software. Box plot diagrams were constructed; the scores of the master trainer were treated as an external gold standard.

Interobserver variability

The coefficient of variation was calculated as described previously.⁷

Intraobserver variability

In April 2005, in Pécs, seven examiners performed the MRSS three times on each of six patients as described above.⁷

Intraclass correlation coefficient

The agreement among the examiners was expressed by the intraclass correlation coefficient (ICC).¹¹ A value of 0.4–0.6 was considered as moderate, 0.6–0.8 as good and >0.8 as excellent agreement.¹⁰

RESULTS

In fig 1, box plots of the three different measurements of the tutors (from Berlin, Budapest, and Vienna, respectively) are presented. In addition, the evaluations performed by European tutors were compared with the single evaluation of a US rheumatologist (DEF) with extensive experience in skin scoring, who was invited to act as the master teacher for the whole course programme.

As shown in table 1, repeated teaching reduced substantially the coefficient of variation for interobserver variability both for this international group of experts (from 50% to 35%) and for Hungarian rheumatologists in a local repeated course (54% vs 32%). With regard to the ICC, both values were good or excellent for experts (0.865 and 0.710, respectively) (for definition of good or excellent ICC, see Methods). The local repeated study indicated that both the interobserver variability and ICC can be substantially improved by repeated teaching. The within-patient SD decreased from 54% to 32%. The ICC increased from 0.496 to the good level of 0.722. The coefficient of variation for intraobserver variability was close to 20% (table 1). When time constraints were placed on the time to complete the examinations as in the second course in Budapest, results were less consistent than in Berlin and Vienna.

For the Budapest course, the box plots of MRSS indicate larger interquartile range values (fig 1B) as compared with the values of the two other courses (fig 1A,C), which suggests better consistency.

When the 90 young rheumatologists performed the skin score measurements, variability was relatively good (35%) due to high numbers, but within-patient SD remained high (5.4; table 1) and the ICC value was low. Similar results were obtained when local repeated courses were organised in Pécs for young rheumatologists (table 1, fig 2). The ICC was good or excellent in all but one case (table 1). The intraobserver within-patient variability was relatively good from the beginning and remained stable thereafter.

DISCUSSION

The aim of this EUSTAR (European Scleroderma Trials and Research group) project was to unify European medical practice in skin score measuring. Overall, the results on MRSS-51 measurements were comparable to two previous independent studies, where the mean (within-patient SD) was 18.3 (4.6) and 17.7 (4.6), respectively.^8,9 Our three investigations for teachers showed similar results (table 1), thereby confirming both that MRSS is an appropriate tool for skin score measuring and that it is possible to teach a large number of rheumatologists the appropriate skin score measuring technique in a short time.

A previous study had demonstrated that the interobserver variability of the MRSS-51 could be reduced by approximately 50%.⁷ The results of our repeat courses likewise demonstrated that the coefficient of variation for interobserver variability can be substantially reduced by repeated teaching (table 1). Although the ICC increased from 0.50 to the good level of 0.72 in the limited local experience with less experienced rheumatologists in Pécs, the ICC of the experts was very good from the beginning, but did not increase (table 1). This underlines the importance of a second training process for less experienced rheumatologists in particular.

It is encouraging that the coefficient of variation for intraobserver variability was around 20%, similar to that found for joint swelling/tenderness counts in rheumatoid arthritis (table 2).¹⁴ Although a low ICC was observed even during the second repeated teaching course, the overall appropriateness of the two repeated courses appeared to be satisfactory (table 1). These findings coincide with previous experience.⁶

When, for the first EUSTAR/EULAR (European League Against Rheumatism) course, almost 100 young rheumatologists were trained, the value of the within-patient SD was relatively good (5.4) with low ICC value, mainly due to the high number of investigators. Interobserver variability after this single teaching course was still considerable, even though experienced experts trained the students. However, the other courses suggest that, following repeated training, interobserver variability could be substantially decreased. After completion, an overall value of 5.5 of within-patient SD for interobserver variability as well as good ICC were achieved in mixed patient populations. This observation is also in agreement with previous experience.⁷

Our study has certain limitations. Even repeated sessions of the experts did not bring them any closer to the main tutor with time (table 1). This may be a limitation of the teaching method, or it could reflect a systematic bias of the principal teacher or a systematic bias of the other experts. The definition of a gold standard remains a critical point in such teaching exercises.

On the other hand, this exercise demonstrates a significant strength—the feasibility of training a large group of relatively inexperienced rheumatologists and the feasibility of a follow-up which completes the training in an efficient and straightforward manner.

In conclusion, in view of a significant amount of observer variability, teaching courses should be repeated at least twice for teaching inexperienced rheumatologists.