Abstract
Objective. Well-designed randomized controlled trials (RCT) mitigate bias and confounding, but previous evaluations of rheumatology trials found high rates of methodological flaws. Outside of rheumatoid arthritis, no studies in the modern era have assessed the quality of rheumatology RCT over time or regarding industry funding.
Methods. We identified all RCT published in 3 high-impact rheumatology journals from 1998, 2008, and 2018. Quality metrics derived from a modified Jadad scale were analyzed by year of publication and by funding source.
Results. Ninety-six publications met inclusion criteria; 82 of these described the primary analysis of an RCT. Over time (1998–2008–2018), trials were less likely to adequately report dropouts and withdrawals (100% vs 82% vs 60%; p < 0.01) or include an active comparator (44% vs 12% vs 13%; p = 0.01). Later trials were more likely to evaluate biologic therapy (11% vs 38% vs 83%; p < 0.01) and report adequate randomization procedures (39% vs 29% vs 60%; p = 0.04). Seventy-nine percent of trials received industry funding. Industry-funded trials were more likely to report double-blinding (86% vs 53%; p < 0.01), patient-reported outcome measures (77% vs 41%; p < 0.01), and intention-to-treat analyses (86% vs 65%; p = 0.04).
Conclusion. Industry-funded trials comprise the majority of RCT published in high-impact rheumatology journals and more frequently report metrics associated with RCT quality. RCT assessing active comparators and nonbiologic therapies have become less common in high-impact rheumatology journals.
Randomized controlled trials (RCT) are the gold standard for assessing the efficacy of pharmaceutical interventions1. Randomization and blinding help mitigate issues of bias and confounding, which limit the validity of observational data2. However, inadequate randomization, poor allocation concealment, and flawed analysis plans may threaten the validity of RCT3,4. Moreover, industry funding has been suggested to bias trials toward a positive result5. Because RCT influence US Food and Drug Administration (FDA) approvals and clinical practice guidelines, their quality is of academic interest.
Previous work assessing rheumatology RCT uncovered low rates of adherence to quality metrics. A 2002 report6 found that fewer than one-third of trials from 1997 to 1998 used adequate randomization, appropriate allocation concealment, or intention-to-treat analysis. A 2003 publication on systemic lupus erythematosus (SLE) trials reported similar results7. Two recent studies on RCT in rheumatoid arthritis (RA) did not observe a bias toward positive outcomes among industry studies and both found higher rates of blinding8,9. One also found fewer than half of studies adequately describing randomization procedures or allocation concealment9.
We describe RCT from 3 high-impact rheumatology journals from the years 1998, 2008, and 2018. All disease states were included. Similar to the methodology employed by Hill, et al, we scored trials using a modified Jadad scale, which assessed the adequacy of important RCT quality metrics6,10. We hypothesized that quality would improve over time and that industry funding would be associated with a higher rate of statistically significant primary outcome measures.
MATERIALS AND METHODS
Journal selection
The top 3 rheumatology journals that publish general interest primary research were identified using the h5-index, which ranks journals according to the volume of highly cited articles over the previous 5 years. A search for “rheumatology” was performed on scholar.google.com/citations?view_op=top_venues11. Annals of the Rheumatic Diseases, Arthritis and Rheumatology, and Rheumatology were included. Nature Reviews: Rheumatology had a high h5-index but was excluded because it does not publish RCT.
Article selection
Articles from 1998, 2008, and 2018 were included if they described an RCT. The following study types were excluded: nonpharmaceutical clinical interventions, open-label extensions, biosimilar equivalence studies, pharmacodynamic/genetic studies, biomarker studies, and dose-finding studies. Author MSP identified potential articles by title, and eligibility was confirmed by review of the abstract and manuscript.
Article review
MSP extracted title, date of publication, and total number of patients included. “Industry funding” was identified and defined as any financial support. MSP also identified whether the article reported a primary or secondary analysis of an RCT, a biologic or targeted disease-modifying antirheumatic drug, an active comparator trial, or a primary outcome with p < 0.05. MSP and AHR conducted a blinded co-review using a modified Jadad scale (Table 1), which retained the elements used by Hill, et al6 (adequate blinding, adequate randomization, allocation concealment, adequate description of withdrawals and dropouts), and incorporated additional elements from the original instrument (identification of a primary outcome, power calculation)10 as well as additional variables (patient-reported outcome included, sensitivity analysis, adjustment for multiple hypothesis testing). The variables and definitions were extracted in a yes/no fashion.
Analysis
Discrepancies between reviewers were adjudicated by conference between MSP and AHR. Pooled interoperator reliability calculated using the kappa statistic (0.66) indicated substantial agreement12. A novel quality scale (normalized from 0 to 10) was constructed from the modified Jadad scale by assigning 1 point for each criterion a study met. Associations between categorical variables were calculated using chi-square testing. The continuous quality scale was compared to year and to journal with an ANOVA model, and to industry funding using the independent samples t-test. All analyses were performed using SPSS version 24.
RESULTS
Out of 3338 articles, 96 were selected for analysis (39 reports in Annals of the Rheumatic Diseases, 18 in Rheumatology, and 39 in Arthritis and Rheumatology). The numbers of publications in each year were 19 (1998), 41 (2008), and 36 (2018). Common conditions included RA (33%), seronegative spondyloarthritis and psoriatic arthritis (14%), osteoarthritis (11%), SLE (9%), vasculitis or myositis (6%), systemic sclerosis (5%), Sjögren syndrome (4%), osteoporosis (4%), and pediatric diseases (3%).
Eighty-two studies were primary analyses of RCT, and 14 were secondary analyses. Among primary analyses (Table 2), articles frequently identified a primary outcome (93%), used an intention-to-treat analysis (82%), and reported a double-blind design (79%). Quality metrics least frequently reported were adjustment for multiple hypothesis testing (16%), performance of a sensitivity analysis (22%), and appropriate reporting of allocation concealment (37%).
Characteristics of primary randomized controlled trials (RCT) in the rheumatologic literature and the proportion that met RCT quality metrics (n = 82). Data are percentages.
Over time (1998–2008–2018), articles reporting primary analyses were less likely to adequately report withdrawals and dropouts (100% vs 82% vs 60%, respectively; p < 0.01), report a p value < 0.05 for the primary outcome measure (80% vs 52% vs 37%; p = 0.02), or compare 2 active therapies (44% vs 12% vs 13%; p = 0.01). This decrease in comparative efficacy research was attributable to changes among industry trials (43% vs 8% vs 11%; p = 0.01). Articles were more likely to evaluate biologic therapy over time (11% vs 38% vs 83%; p < 0.01) and to report adequate randomization procedures (39% vs 29% vs 60%; p = 0.04). The majority of trials received industry funding (79%); changes over time were not statistically significant (78% vs 74% vs 87%; p = 0.43). Industry-sponsored trials were more likely to evaluate a biologic therapy (57% vs 18%; p < 0.01) and to report double-blinding (86% vs 53%; p < 0.01), patient-reported outcome measures (77% vs 41%; p < 0.01), or intention-to-treat analyses (86% vs 65%; p = 0.04).
The average score on the quality scale was 6.56. There were no significant differences in the overall quality among journals (6.18 for Annals of Rheumatic Diseases, 5.19 for Rheumatology, and 6.52 for Arthritis and Rheumatology; p = 0.14). The quality score increased numerically by year (6.05, 6.50, 6.93), but this was not significant (p = 0.39). Industry funding was associated with a higher score (6.75 vs 5.82 for non-industry; p < 0.01). Secondary analyses of RCT had a lower quality score than primary analyses (3.65 vs 6.56; p < 0.01). There were no significant differences in quality between studies that reported a p value < 0.05 (6.50 vs 6.80; p = 0.51), studies on biologic therapies vs nonbiologic therapies (6.18 vs 6.09; p = 0.85), or on studies with more or less than 200 patients (6.67 vs 5.92; p = 0.16).
DISCUSSION
Over time, high-impact rheumatology journals published more RCT that assessed biologic therapy and performed adequate randomization. Adequate description of withdrawals and dropouts, statistically significant primary endpoints, and comparative effectiveness research became less frequent. Industry trials were more likely to use double-blinding, patient-reported outcome measures, and intention-to-treat analysis, and had a significantly higher overall quality score than non-industry trials.
These results extend previous work to a broad range of disease states and the modern era6,9. Both overall quality and randomization have improved over time. Contrasting with these positive trends, fewer trials adequately described dropouts and withdrawals. This makes it difficult to assess the risk of attrition bias13. Surprisingly, fewer trials met their primary endpoint over time. This may be attributable to requirements that trials be listed on ClinicalTrials.gov, which may have encouraged the publication of negative studies14,15.
This investigation confirms a high rate of industry sponsorship of RCT. As with RA trials, industry sponsorship was associated with higher rates of adequate blinding and intention-to-treat analysis9. Industry-sponsored trials also included patient-reported outcome measures at a higher rate16. Similar to studies in RA, we did not observe a significant association between industry funding and positive primary outcome measures8. This association has been observed in other fields of medicine17, and our study may have been underpowered to detect a meaningful difference.
The decline in publication of comparative efficacy research — from nearly half of trials in 1998 to one in 8 by 2018 — appears to be driven by industry funding. This may be appropriate early in drug development, but clinicians need trials comparing the relative effectiveness of therapies to guide clinical decision making. Such trials have started to emerge in RA18,19, but remain lacking for many disease states. Moreover, in many conditions, off-patent therapies without industry sponsorship have never been compared20. If industry funding and incentives drive our research agenda, valuable questions may remain unanswered.
This report has a number of limitations. The exclusion of high-impact general medicine journals and lower-impact rheumatology journals may bias our results. Although we assessed almost 100 articles, this study may be under-powered to detect significant differences over time or between funding sources. Finally, the reporting of quality metrics may not reflect actual practice21, and other important quality metrics, such as discrepancies between FDA analysis and published analysis22, were not included.
Trends toward improvement were observed in multiple specific metrics over time. Studies that received industry funding had higher rates of individual quality metrics and higher quality overall. The decrease of comparative efficacy research and its effect on the practice of rheumatology deserves further investigation.
- Accepted for publication March 11, 2020.
