Abstract
Objective. To examine trends in methodological quality of studies of endstage renal disease risk in lupus nephritis, 1970–2015.
Methods. We assessed quality using the Newcastle-Ottawa scale for observational studies, and the Cochrane Collaboration’s risk-of-bias tool for trials.
Results. In observational studies, description of enrollment criteria was high but decreased over time. Adequacy of followup was low but improved. Inception cohorts and community-based studies were uncommon. Trials had low risk of bias in blinding and selective outcome reporting but most had unclear risks in sequence generation and allocation concealment.
Conclusion. Methodological quality was mixed, with limited improvement over time.
Endstage renal disease (ESRD) is a severe potential complication of systemic lupus erythematosus. To examine whether the risk of ESRD changed over time, we recently performed a systematic review and metaanalysis of all published studies up to April 2015 that reported the risk of ESRD in lupus nephritis1. We observed that risks of ESRD improved between the 1970s and the mid-1990s and then plateaued, with an increase in the 2000s.
Assessing the methodological quality of clinical studies is important for proper interpretation of their findings2. With the maturation of clinical epidemiology since the 1980s and the development of reporting guidelines, such as the Consolidated Standards of Reporting Trials (CONSORT) statement for clinical trials3 and the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement for observational studies4, both the design and reporting of clinical studies would be anticipated to have improved over time. Several tools have been proposed for assessing study quality. In 2008, the Cochrane Collaboration introduced a tool for assessing the risk of bias in clinical trials5. Wells, et al proposed the Newcastle-Ottawa scale for assessing the quality of nonrandomized studies in metaanalyses6.
The purpose of our current study was to determine trends in the methodological quality of studies of the risk of ESRD in lupus nephritis since 1970.
MATERIALS AND METHODS
In our systematic review, we searched PubMed, Embase, and the Cochrane Database of Systematic Reviews from their inceptions to April 7, 2015, for studies on the risk of ESRD in adults with lupus nephritis1. We included 187 studies: 144 observational studies and 43 clinical trials (Supplementary material, available from the authors on request). The study was exempted from ethics review by the US Office for Human Research Protections because it was designated as research not involving human subjects.
Methodological quality assessment was independently performed by each author using a protocol. Discrepancies were resolved by discussion. For observational studies, we assessed methodological quality using 10 items in 3 domains, adapted from the Newcastle-Ottawa scale6: completeness of description of enrollment criteria and treatment, adequacy of followup, and representativeness of the cohort (Table 1). In this analysis, we omitted 2 items (related to renal biopsy and patients with chronic kidney disease) because they are not generalizable measures of quality applicable to studies of nephritis outcomes. Each item was rated as present or absent.
For clinical trials, we used the Cochrane Collaboration’s tool5 to assess the risk of bias in 5 domains (Table 1): sequence generation; allocation concealment; blinding of participants, personnel, and outcome assessors; incomplete outcome data; and selective outcome reporting. In each domain, the risk of bias was graded as high (plausible bias that seriously weakens confidence in the results), uncertain (plausible bias that raises some doubt about the results), or low (plausible bias unlikely to seriously alter the results).
We classified studies by their decade of publication (pre-1990, 1990–1999, 2000–2009, and 2010 and later) and computed the proportion of studies in each decade that satisfied each quality measure. For observational studies, we used logistic regression analysis to compare these proportions by examining linear trends in OR across decades. Because large studies and studies from developed countries may have designs different from small studies and those from developing countries, we adjusted these analyses for study size (fewer than 100 patients vs 100 or more patients) and developed versus developing country of origin. Development status of the study’s country of origin was based on the World Bank’s rating7. For trials, we compared trends across decades using chi-square tests. There were too few trials to support multivariate analysis.
For both sets of studies, we also examined trends using publication year as a continuous variable in logistic regression models, and tested whether quality reporting differed after publication of the STROBE and CONSORT statements.
We used SAS programs, version 9.3 (SAS Institute) for analyses.
RESULTS
Observational studies
Twelve studies were published before 1990, 36 studies were from 1990 to 1999, 55 studies were from 2000 to 2009, and 41 studies were published in 2010 or later. Overall, 38% were from developing countries, and 32% were large studies. The proportion of observational studies from developing countries increased from 17% in the pre-1990 period to 46% in 2010 and later. The proportion of large studies also increased over this time from 8% to 39%, with large studies evenly reported from developed countries and developing countries (50% each).
Description of enrollment criteria was high, but decreased from 92% before 1990 to 78% in 2010–present (p = 0.04; Figure 1A). The adjusted OR for reporting of enrollment criteria, with pre-1990 as the reference group, were 0.82 (0.08,8.82) for 1990–1999, 0.27 (0.03, 2.38) for 2000–2009, and 0.23 (0.02, 2.06) for 2010 and later. Reporting on renal function and treatment at study entry was high and stable over time.
Top panel: Proportion of observational studies that met study quality criteria by decade of publication for ACR criteria, renal function, and treatment. Bottom panel: Proportion of observational studies that met study quality criteria by decade of publication for community-based and inception cohorts. ACR: American College of Rheumatology.
The proportion of studies of inception cohorts ranged from 33% to 43% and did not change over time (p = 0.43). Notably, in studies from developing countries this proportion increased from 0% before 1990 to 40% in 2010–present. The proportion of studies with community-based enrollment remained very low (Figure 1B).
The proportion of studies reporting on losses to followup was low but improved across decades (Figure 2). The proportion that described the nature of losses increased over time (p = 0.05), with an adjusted OR for reporting on losses to followup in 2010 and later of 5.15 (0.57, 46.21), compared to pre-1990. Similarly, the adjusted OR for a study having < 20% loss to followup in 2010 and later was 3.54 (0.65, 19.11), compared to pre-1990 (p = 0.06). There was no appreciable increase in the proportion of studies reporting results with Kaplan-Meier plots. Seventy percent of recent studies did not use the Kaplan-Meier method to assess the time to ESRD.
Proportion of observational studies that met study quality criteria by decade of publication for Kaplan-Meier methods and loss to followup.
The median number of criteria (of 8) met per study in studies from developing countries increased over time (2.5, 3, 3, and 4 in pre–1990, 1990–1999, 2000–2009, and 2010 or later, respectively), but did not increase in studies from developed countries (3.5, 4, 3.5, 4 in each decade, respectively).
Using publication year as a continuous variable, only the likelihood of reporting renal function increased over time (Supplementary Table 1, available from the authors on request). There was no change in likelihood of quality measures being included after the STROBE statement in 2007 (all p > 0.15).
Clinical trials
Six trials were published before 1990, 10 from 1990 to 1999, 15 from 2000 to 2009, and 12 in 2010 or later. Most trials had an unclear risk of bias for sequence generation and allocation concealment, increasing to 80% and 92%, respectively, during the last decade (Figure 3). This principally was the result of incomplete description of the methods of randomization. All trials were rated as having low risk of bias in the domains of blinding and selective outcome reporting. Although most trials were not blinded, the objective nature of the outcomes, including primarily laboratory measures and ESRD or death, afforded a low likelihood of bias. Additionally, 80%–90% of trials had a low risk of bias for incomplete data on outcomes, with no trend across decades. The median number of criteria (out of 5) with low risk of bias per study was 3 in each decade. There were no trends across calendar years of publication (Supplementary Table 2, available from the authors on request), and no change after the CONSORT statement in 1996 (all p > 0.65).
Risk of bias among clinical trials in 5 domains, by decade of publication.
DISCUSSION
Among observational studies of ESRD risks in lupus nephritis, at least half of the methodological quality criteria were not commonly met. Only 2 criteria, both related to losses to followup, improved significantly over time. The quality of observational studies from developing countries improved over time, coming to match those from developed countries in number of criteria satisfied in recent years. Among trials, blinding, outcome reporting, and completeness of outcomes had low risk of bias, while incomplete descriptions of the randomization process led to unclear risks of bias for these criteria.
Several factors may account for these findings. Lack of appreciation of the study design features that are important for ensuring high methodological quality may cause investigators to omit implementation of these characteristics or omit them in study reports8,9,10. Reporting is essential to ensure that high-quality studies are recognized. Some decisions that occur at the study design stage, particularly the choice to enroll a community-based or inception cohort, are often predicated on feasibility10. Even though such studies are more difficult to conduct, their enhanced validity should encourage greater use.
Several key design features should be included in all outcome studies. Enrollment criteria, patient characteristics, and treatment should be reported. Kaplan-Meier curves should be used to present time-to-event data. Also, data on the adequacy of followup, including the proportion and reasons for losses to followup, should be reported11. Inception cohorts should be studied whenever possible. In trials, the process of random allocation sequence and adequate allocation concealment should be reported to allow assessment of possible selection bias12,13.
While many studies of outcomes in lupus nephritis fulfill several quality criteria, there is room for improvement. Researchers can improve the quality of the literature by considering these features in both the design and reporting of studies. Readers should also consider how the presence or absence of these features affect their appraisal of a study’s findings.
Footnotes
The primary funding source for this work is the Intramural Research Program, NIAMS.
- Accepted for publication January 4, 2017.
