Reappraisal of OMERACT 8 Draft Validation Criteria for a Soluble Biomarker Reflecting Structural Damage Endpoints in Rheumatoid Arthritis, Psoriatic Arthritis, and Spondyloarthritis: The OMERACT 9 v2 Criteria

RESULTS AND DISCUSSION

Objectives of the validation process

The following primary objectives for an RA biomarker were proposed by the biomarker working group in London and discussed at OMERACT 9.

• A. Change in the biomarker reflects/predicts change in the damage endpoint at the level of:

  • Group: The biomarker constitutes an endpoint for clinical trials and cohort studies.

  • Individual patient: The biomarker constitutes an endpoint for clinical practice.

• B. Change in the biomarker reflects/predicts change in the damage endpoint independently of known predictors, e.g., Disease Activity Score, baseline damage, rheumatoid factor, anti-cyclic citrullinated peptide, shared epitope, C-reactive protein (CRP)/erythrocyte sedimentation rate.

The primary objectives for a PsA biomarker were proposed to be the same as for RA. The following primary objective was proposed by the group for an AS biomarker: Change in biomarker predicts change in damage endpoint independently of known predictors, e.g., baseline damage.

No revisions to these objectives were proposed at the OMERACT 9 meeting. In retrospect it was recognized that the OMERACT Filter criteria of truth and discrimination largely overlap in the setting of validation of predictive biomarkers; this is in contrast to the setting of validation of evaluation measures, where the Filter was originally developed⁴. The general question attached to the element of Truth is: “does it measure what it’s supposed to?” and in the setting of prediction this translates to: “Does it adequately predict what it should predict?” Formulated in this way, the question is very similar to that posed for the element of Discrimination: “Does the measure distinguish between states of interest?” If the biomarker is modifiable (e.g., a soluble biomarker) and change in this biomarker over the course of a longitudinal study or clinical trial is shown to consistently parallel change in radiographic progression as disease activity changes and/or with changes in therapy, it can be stated that the biomarker reflects damage to the extent that it no longer becomes necessary to measure radiographic progression. To be clinically useful, such a biomarker should reflect radiographic progression in groups of patients assessed in longitudinal studies and clinical trials as well as in the individual patient, thereby allowing the clinician to manage therapy according to prognostic risk.

Properties of a clinically useful RA or PsA biomarker (Figure 1)

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

The properties of an “ideal” RA biomarker that can substitute for radiographic damage in clinical studies and individual patients with RA. The relationship between levels of the biomarker, Disease Activity Score (DAS), and the interval change in radiographic severity over 3 months (radiographic progression) is illustrated. Tx: therapy.

Change in the damage endpoint can occur relatively rapidly in RA and has been documented as soon as 3 months in patients in a placebo section of a trial⁴. To be clinically useful, an RA or PsA biomarker should be a leading indicator of change in the damage endpoint, and the change in biomarker should correlate with the interval change in damage progression. Moreover, the magnitude of the change in biomarker level should consistently reflect the subsequent degree of change in radiographic progression, whether this is associated with spontaneous or treatment-induced change in disease activity. Since the biomarker should reflect radiographic progression independently of known predictors, such as the Disease Activity Score (DAS), changes in biomarker levels may not necessarily correlate with changes in other predictors, and the magnitude of change may vary in relation to change in other predictors. However, from the perspective of clinical utility it would be desirable if such a biomarker was more responsive than routinely assessed clinical and laboratory measures associated with radiographic progression. Finally, measurement of the biomarker should add prognostic information regarding radiographic progression over and above the combined information obtained from all other known predictors. These properties should not only be demonstrable at the group level, as in the evaluation of response to treatment in clinical trials and cohort studies, but also at the individual patient level. Figure 1 illustrates these principles in this hypothetical example of a clinically useful RA biomarker. At baseline, biomarker level, DAS, and radiographic progression are high. Treatment with methotrexate is associated with an immediate reduction in biomarker level in the first 3 months that is much greater than the reduction in DAS and precedes any change in radiographic progression that decelerates from 3–6 months. From 6 months there is a very large increase in biomarker level that precedes (i.e., predicts) a large increase in radiographic progression observed after 9 months. At the one-year point the anti-tumor necrosis factor (TNF) treatment is added to methotrexate and this is associated with an immediate and very large decrease in biomarker level from one year to 15 months that precedes the large reduction in radiographic progression observed from 15 months. Note that the magnitude of the change in the RA biomarker level reflects the subsequent change in the degree of radiographic progression and that the biomarker is also much more responsive than the DAS.

Properties of a clinically useful AS biomarker (Figure 2)

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

How a “response-to-treatment” effect of an AS biomarker can provide leading indicator prognostic data for radiographic damage in AS. Treatment 2 reduces biomarker levels at 6 months to a greater degree than treatment 1, which predicts less radiographic progression over 2 years with treatment 2.

Validation of a biomarker for radiographic progression in AS presents different challenges because radiographic progression is slow, requiring at least 2 years of followup before progression can be reliably measured, because there is as yet no demonstrable association between disease activity measures and radiographic progression, and because there is currently no good evidence that any treatment slows progression in this disease^5,6. An AS biomarker that predicts damage progression may, therefore, vary unpredictably over the long time-course of observation. Analyses that address this variability over the duration of followup, for example area under the curve, may still define an important association between the biomarker and damage progression in cohort studies. However, the conclusions from this type of analysis may not be useful to the clinician managing the individual patient. Assessment of the predictive validity of a change in biomarker level early in the course of followup is clinically meaningful, although this may be problematic when the factors that govern progression are largely unknown, and biomarker levels may fluctuate unpredictably. This challenge might be addressed in a “response to treatment” study design using an intervention that has been shown to induce a consistent and persistent change in the biomarker, e.g., reduction in matrix metalloprotease-3 induced by anti-TNF agents⁷. The magnitude of such a change should reflect the subsequent change in risk for radiographic progression and could be examined at both the group and the individual patient level. This approach would require preliminary studies addressing the influence of treatment on the biomarker. Figure 2 illustrates these principles in this example of a clinically useful AS biomarker in 2 groups of patients that have the same radiographic progression at baseline but are exposed to 2 different treatments, with treatment 2 having a greater effect on the AS biomarker than treatment 1. The greater reduction in the level of the AS biomarker with treatment 2 predicts a greater reduction in radiographic progression.

Reappraisal of OMERACT 8 draft criteria

The following conclusions were reached by the group at the London meeting in developing a revised criteria set.

1. Voting for inclusion/exclusion of individual criteria was decisive in that consensus was achieved after the first round of voting for all the criteria. Nevertheless, even those criteria that were voted for exclusion were still considered desirable in developing a body of scientific evidence in support of a biomarker as reflecting joint damage. Consequently, the group proposed that the criteria be divided into 2 categories, an essential or core set and a desirable but nonessential category (see Appendix).

2. The 5 criteria organized under the category of truth were considered desirable but not essential criteria. The original wording of Criterion 2 was revised to remove the word “immunohistochemically” in describing localization to joint tissues. The group felt that there are biomarkers that may be useful but have not been specifically localized to joint tissue using immunohistochemistry.

   Conclusions are often drawn too early about the biology and mechanism of action of biomarkers from work conducted in animal models. This can lead to misinterpretation of the data in humans. Animal models may be useful for understanding pathophysiology but are not useful as a validation tool. A biomarker may be localized to joint tissues by several different methodologies of which immunohistochemistry is one example. But few demonstrate specificity for target of joint tissue origin and some clinically useful biomarkers are not primarily of joint origin, e.g., CRP. Requiring an understanding of the relation between a biomarker and turnover of joint tissue components is problematic, because there is no recognized in vivo “gold standard” for bone/cartilage turnover and in vitro/ex vivo models of bone/cartilage turnover are poorly predictive of in vivo relationships. In vivo techniques are also invasive and therefore not feasible. With respect to demonstrating correlations with other surrogates reflecting damage endpoints this criterion might be appropriate if magnetic resonance imaging is used, because increasing data supports its predictive validity for radiographic damage endpoints. It would not be appropriate for other biomarkers such as CRP because of weak association with damage.

3. Under the category of discrimination, Criterion 6 was revised as follows: “The assay for measurement of the biomarker is reproducible (coefficient of variation: intra-assay less than 10%, inter-assay less than 15%)” was changed to “inter-assay less than 10%,” as it was felt to be a more stringent cutoff.

4. Criterion 7 describing sources of variability on levels of the biomarker was revised to stipulate core variables that should always be examined during the validation process (age, sex, ethnicity, circadian rhythms, body mass index, renal/hepatic function, fasting/nonfasting) and desirable but nonessential variables (menopause, comorbidity, physical activity, nonsteroidal antiinflammatory drugs). This criterion was considered important but lacked feasibility in its original form because of its complexity in requiring examination of so many variables.

5. Criteria 8 and 9 under the category of discrimination referring to metabolism of the biomarker and sensitivity/specificity of the biomarker in the disease population compared to healthy controls, respectively, were considered desirable but nonessential.

   Metabolism, clearance, and half-life of a biomarker are not relevant with respect to its predictive validity because what is being tested is the strength of the association between change in levels of a biomarker and change in damage endpoints. Its pharmacokinetic behavior in healthy and affected individuals is, therefore, not relevant to its validation for this objective. Specificity is also not relevant if the biomarker is being used as a predictor of outcome in affected individuals and is of greater importance if the purpose of the biomarker is diagnostic.

6. Criteria 10, 11, and 12 require that the biomarker demonstrates independent association with the radiographic endpoint in patients in (i) prospective cohort studies (Criterion 10); (ii) randomized controlled trials (Criterion 11); and (iii) pre-radiographic disease (Criterion 12). The 3 criteria were combined to state that the association be evident “at all stages of disease at the level of both absolute and relative change in prospective cohort studies and randomized controlled trials (except for AS) of adequate sample size, and followed for a sufficient duration to detect change in x-ray damage score.” This was also modified in recognition that validation of an AS biomarker in a randomized controlled trial would not be feasible using a plain radiographic endpoint in view of the necessity to conduct a 2-year trial.

7. Criterion 13, under feasibility, was modified to delete the requirement for methodological simplicity and to emphasize accessibility of the biomarker assay. International standardization of an assay was thought to be a difficult but desirable criterion. The statement was further modified to remove the criterion for the assay to be “well characterized” as this was considered vague.

8. Criterion 14 was modified to address the requirement for stability after repeat freeze/thaw cycles and after longterm storage (> 1 year).

   Several additional revisions to the criteria were proposed at the OMERACT 9 meeting. For Criterion 6, addressing assay reproducibility, it was recommended that this be conducted through formal reliability analysis. For Criterion 7, describing sources of variability, the importance of using appropriate controls as compared to merely healthy individuals was proposed to address demographic confounders. For Criterion 13, addressing accessibility of the biomarker assay, the particular relevance of this to clinical practice was stated. A single revision was proposed to one of the nonessential criteria addressing the metabolism of the biomarker (Criterion 8), where the stipulation was added that the impact of concomitant therapy should be determined. The percentage of OMERACT 9 participants attending the workshop who voted “yes” on the question of agreement with this framework and the specific criteria was 79%.