Proposal for Levels of Evidence Schema for Validation of a Soluble Biomarker Reflecting Damage Endpoints in Rheumatoid Arthritis, Psoriatic Arthritis, and Ankylosing Spondylitis, and Recommendations for Study Design

RESULTS AND DISCUSSION

Levels of evidence framework

The generic framework for a levels of evidence scheme was adapted for soluble biomarkers at the London meeting (see Appendix), and following modification at OMERACT 9 was accepted by 85% of workshop participants (Table 1). Agreement was reached on the following adaptations to the domains:

1. Target outcome domain. The grading of 0 (disease-centered, reversible) to 5 (death) was not considered relevant to the validation of a biomarker reflecting structural damage. A grading of 0 to 3 [patient centered, irreversible, minor organ/clinical morbidity (radiography)] was considered appropriate, with radiography being accepted as a patient-centered outcome. Some argued that the target outcome has already been defined as radiography in formulating the principal objectives of the validation process and that there is, therefore, no need to include this domain in the scheme. The counter-argument was that other measures of damage, e.g., magnetic resonance imaging (MRI), may be increasingly relevant as validation data increase. For example, it has been shown that bone marrow edema on an MRI has predictive validity for radiographic damage and can be reliably detected and quantified⁵. As clinicians increasingly target and require guidance in the treatment of pre-radiographic disease, MRI may increasingly constitute a relevant outcome for biomarker validation studies.

2. Study design domain. The grading of 0 (animal studies, case reports, cross-sectional, retrospective) to 5 (≥ 3 RCT each of different drug class, ≥ 3 randomized surrogate objective trials) was modified to incorporate an equal weighting for RCT and prospective observational studies. Longitudinal studies would have to be consistent with the minimum standards for longitudinal study design advocated by the group (see below). Randomized surrogate endpoint trials were considered too high a hurdle for the objectives of this biomarker validation process. The proposed ranking recognizes the importance of biomarker validation with different drug classes. For example, it has now been consistently demonstrated that C-reactive protein has predictive validity for structural damage in patients with RA receiving methotrexate, but not in those receiving anti-tumor necrosis factor therapies^6–8. Both longitudinal cohort studies and RCT are deemed essential. The former address validation in a wider spectrum of patients and over longer time periods, while the latter can more readily address validation with different drug classes and potential confounders.

3. Statistical strength domain. In evaluating the association between marker change and target change or the prediction of the effect of treatment on marker change and target change, regression-based modeling is the primary statistical technique, and the fitting of the model will lead to a goodness of fit statistic (such as, percentage of the variation explained by the model R²). The statistical evidence of the association or prediction can be determined using a modification of the Sterne and Smith interpretation of the p value, taking the number of observations into consideration⁹, whereas the statistical strength per se can be based on the model, with the effect size determined using the coefficient for the biomarker [e.g., the odds ratio (OR)]. This effect estimate can be translated using Cohen’s standardized mean difference (SMD), i.e., an OR value can be transformed into an SMD¹⁰: \batchmode \documentclass[fleqn,10pt,legalpaper]{article} \usepackage{amssymb} \usepackage{amsfonts} \usepackage{amsmath} \pagestyle{empty} \begin{document} \[ \mathit{\hbox{ SMD }}=\surd 3/\pi \hspace{0.17em}\hspace{0.17em}\hbox{ log }OR \] \end{document} and levels of strength can be derived based on the usual thresholds for interpreting “fair” (0.2), “good” (0.5), “very good” (0.8).

4. Penalties domain. The grading in the generic template proposal was largely adopted although with the stipulation that rather than being additive for different studies, the highest score would be applied as a penalty and that the same minimum standards be applied to the evaluation of study design.

5. Performance domain. This domain is not a component of the generic template but it was agreed that biomarkers should have been validated according to the criteria comprising this domain before proceeding with clinical validation studies. The criteria address standards of reproducibility, feasibility (readily accessible, availability of international standards, costs), biomarker stability, and evaluation of confounders that are defined in the OMERACT 9 biomarker validation draft criteria under the categories of feasibility and discrimination (criteria 1, 2, 4, and 5).

Longitudinal study design consensus

The following design issues and key recommendations were highlighted at the London meeting for consideration in the Delphi voting exercise: principal inclusion criteria, study design (RCT vs observational), treatment strategy, study duration, appropriate damage endpoints, frequency of assessment, and sample collection and processing. A total of 52 ASAS and 45 GRAPPA members provided additional input into the items proposed for the AS and PsA Delphi exercises, respectively. For the first round there were 130 OMERACT 9 participants, 46 ASAS members, and 53 GRAPPA members who participated in the Delphi voting exercise for RA, AS, and PsA, respectively. In the second round of voting, the corresponding number of participants was 113, 43, and 46, for OMERACT 9 participants, ASAS, and GRAPPA members. The final results of these 3 Delphi exercises are presented in Table 2.

Failure of consensus was evident for 2 key items that were discussed further at OMERACT 9. The first focused on the diagnostic inclusion criterion for a validation study of an RA biomarker. There were 2 principal schools of thought on this matter. Some considered it desirable for a validation study, especially the first, to stipulate the American College of Rheumatology (ACR) classification criteria on the premise that such patients would be not only relatively homogeneous but also more likely to demonstrate disease progression, which increases statistical power. Inclusion of patients with a wide spectrum of disease activity and severity was also considered desirable, since some biomarkers may reflect radiographic progression better in early versus late disease and vice versa. Other participants were supportive of differentiating patients on the basis of the anti-cyclic citrullinated peptide (CCP) antibody test in early disease on the premise that these patients are a distinct group both prognostically and on the basis of pathophysiology^11–13. The latter could, therefore, imply quantitative and/or qualitative differences in the relationship between a particular biomarker and radiographic damage. A compromise proposal was to include patients on the basis of the ACR criteria but then to prespecify analysis stratified by anti-CCP status. Both RCT and observational studies were considered equally desirable to ensure generalizability of study findings, although RCT for AS were not considered feasible because progression of radiographic change is not reliably detected prior to 2 years in patients on standard therapy¹⁴. Validation in studies employing diverse and flexible treatment strategies was considered desirable since the real clinical utility of a biomarker is dependent on the demonstration that levels of the biomarker are independently associated with structural damage regardless of treatment approach.

Consensus was not achieved in regard to the minimum standards for the handling of biomarker samples because a substantial minority of respondents refrained from voting in the Delphi exercise as they assigned themselves the designation “not an expert.” It was decided by consensus that the biomarker group should develop a proposal for the systematic handling of biomarker samples, which is presented in Table 3. First, the group has recommended the collection of both urine and serum. Although feasibility is an obvious advantage for serum, it is important to standardize the collection of serum in view of previous reports that preanalytical handling of serum influences certain biomarker levels, such as metalloproteinases, which are released from platelets and leukocytes particularly when using collection tubes that enhance clotting (kaolin-coated)^15,16. Ideally, possible interfering factors should be identified as discussed under the Performance criterion assay-related confounders (Table 1) and recommendations for standardization of sample collection clarified prior to clinical studies. A practical problem is that several biomarkers are often tested simultaneously, and collection procedure may not be optimal for all biomarkers. In addition, samples are often collected as a routine during observational studies and RCT and then analyzed retrospectively. It is obvious that analysis of the individual biomarkers should only be done on samples that have been obtained and handled in a way that ensures reliable measurement with respect to diurnal variation, centrifugation, freezing temperature, stability to freeze/thaw cycles, etc. These characteristics may vary considerably from marker to marker. It will not always be known in advance which markers will later be analyzed. Therefore, a default approach is to recommend standardized operating procedures for sample collection as outlined in Table 3.