MATERIAL AND METHODS

Table 1 shows the three preliminary activity indices established in our previous study.² For the external validation of the indices, the protocol management members (GV, SB, SD’A, ADR) stratified the dataset in three subgroups with increasing activity (inactive, moderately active, and active/very active). Thirty charts (10 out of each group) were then randomly selected and sent to four experts (CMB, LC, HN, PGV) identified as representative of different countries and geographical areas and available to give their evaluation at the time the study was carried out. The four experts were requested to rank the charts under blinded conditions from 1 to 30 (least active disease to most active disease).

We then investigated the correlation between the ranks given by each expert and the respective activity indices for each patient using a Spearman rank correlation. In our previous work we had shown that a score of over 3 was the best single point discriminator between active and inactive disease. We then compared the median rank, for each expert, between those with and without active disease using the Mann-Whitney U test.

RESULTS

Table 2 shows the correlations between the disease activity rankings given by each of the four experts and the three EScSG indices. A significant correlation was found between the ranks given by each of the four external investigators and the derived index for the whole series. However, the correlations were not uniformly significant for the dcSSc index and very rarely significant for the lcSSc index. Therefore, we decided only to consider the whole series index for further evaluation.

Table 3 shows the correlations between the ranks given by the four experts for the 30 charts. This table clearly shows that even among experts in the field there is not a very high level of agreement in the assessment of disease activity.

Figure 1 shows the distribution of the ranks given by each expert in patients with a whole series index ⩽3 and >3 (that is, the value that had been found, in our previous paper, to discriminate between patients with low and high activity). The ranks in patients with high activity disease were in each case significantly higher than those given in patients with low activity.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Distributions of the ranks given by each investigator in patients with active and inactive disease according to the whole series index.

DISCUSSION

We identified and validated three sets of criteria devoted to defining disease activity in SSc as whole, in dcSSc, and in lcSSc. Our first validation, however, was carried out by the jackknife approach,³ which is affected by an intrinsic circularity. In this report we provide the results of an external validation process. Our data show that the index derived for use in all patients is the only one which was validated by all the external investigators. We are inclined to think that the unsatisfactory performance of the dcSSc index may be due to the small number of patients investigated, and the unsatisfactory performance of the lcSSc to the difficulties in identifying activity in patients whose disease has an indolent course. Therefore, further evaluation of these subgroup indices should be undertaken.

To define disease activity criteria for SSc, we used the same methodology previously used by two of us (SB and WB) to define both classification criteria for Sjögren’s syndrome⁵ and activity criteria for systemic lupus erythematosus, which has been validated both internally⁶ and externally.⁷ Therefore we relied on real patients. Such methodology had the substantial advantage of being based on parameters which are currently evaluated in each centre, enhancing the feasibility of the developed index.

In the present study, as well as in our first study, the correlation between the activity items and the “gold standard” (that is, the activity score, from 1 to 10, given in blinded conditions by protocol management members²) was lower than that detected in other diseases such as systemic lupus erythematosus.⁶ This might suggest that the developed index has an intrinsic weakness. However, when a correlation between the ranks given by different experts was carried out (table 3), a similar level of agreement was detected. We therefore suggest that the low level of agreement found reflects the low level of concordance on the degree of disease activity in SSc. Defining activity in SSc is different from defining activity in other diseases such as rheumatoid arthritis and systemic lupus,^6,8–10 because of the lack of clear flares and quiescent phases and its indolent course.

The presence of three Δ factors based on patient self assessment may cause some concern because of the lack of standardisation. However, such an approach has been accepted as valid and reliable in assessing disease activity in rheumatoid arthritis.^10,11

In conclusion, we have developed a validated index to assess disease activity in SSc. This index must still be considered preliminary. However, it should be viewed as the first necessary step towards developing a better agreement among different investigators in the assessment of disease activity in SSc. It can be used as a preliminary activity index in clinical investigational studies. Nevertheless, the search for other measures of disease activity in SSc must continue.