On the basis of metaanalysis of randomized studies assessing the symptoms and radiological progression of patients with osteoarthritis (OA), investigators have concluded that health authorities and health insurers should not cover the costs of glucosamine and chondroitin, and new prescriptions to patients who have not received treatment should be discouraged. This conclusion has the potential to change current management of OA mainly in Europe, where these compounds are prescribed drugs. But should metaanalysis be considered the ultimate level of evidence and sole support for these conclusions? This editorial addresses this question.
A metaanalysis is defined as a panel of statistical methods of combining data coming from a set of comparable studies addressing a particular question. A metaanalysis may or may not be a part of a systematic review yielding a quantitative summary of the pooled results1. In general, metaanalyses are used to support evidence-based recommendations. The general aim of a metaanalysis is to more powerfully estimate the true “effect size” as opposed to a smaller effect size derived in a single study under a given single set of assumptions and conditions. Reasons for considering a metaanalysis in a review include increasing the power (higher chance to detect an effect) and precision to answer questions not posed by individual studies, and to settle controversies arising from apparently conflicting studies or to generate new hypotheses. Of course, the use of statistical methods does not guarantee that the results of a review are valid, any more than it does for a primary study.
It has been largely recognized that good research practices in conducting metaanalyses and interpreting the results are indispensable, and like any tool, statistical methods can be misused2,3. There are some essential prerequisites for a metaanalysis to be valid. Briefly, all studies included must fulfill prespecified criteria, address a set of related research hypotheses, and have the same or closely comparable study populations, methods and procedures, and the data must be complete and free of biases. The initial question for doing a clinically meaningful metaanalysis is how similar must the studies be in order to be included — the more similar the studies, the more valid the metaanalysis. A less restrictive requirement of similarities between the studies allows more trials to be included into the metaanalysis, which makes it easier to achieve statistical significance. However, this can decrease the reliability of the conclusions. Moreover, metaanalyses performed by strong advocates of a particular position in an ongoing controversy are at higher risk for bias.
Systematic reviews and metaanalyses are widely used in the biomedical sciences and are being published with increasing frequency in biomedical journals of almost all specialties. Moreover, metaanalyses are important evaluations from a public health perspective, not only to develop clinical practice guidelines, but also to assist policy makers and healthcare professionals in using their findings for decision-making4. However, the validity and clinical relevance of the conclusions of such studies cannot be taken for granted. This is the case in a recent network metaanalysis from Wandel, et al5 in which it is concluded that glucosamine and chondroitin, alone or in combination, do not clinically improve joint pain in patients with OA of the knee or hip or have an influence on narrowing of joint space. Also, a final strong conclusion was drawn that “health authorities and health insurers should not cover the costs of these preparations, and that new prescriptions to patients who have not received treatment should be discouraged.”
The controversy generated shortly after the publication of this article in the British Medical Journal, as reflected by several letters to the editor posted on the journal’s website (some of which were also published in the printed edition)6,7,8 as well as opinion articles and critical commentaries9,10, merits a few comments. Several expert groups in the field of OA have questioned the validity of the conclusions for different reasons, among them: (1) the use of a complex Bayesian approach and arbitrary calculation of the effect size, prespecified as a minimally clinically important difference of 0.37 SD units, corresponding to 0.9 cm on a 10-cm visual analog scale, whereas there is no universally accepted definition of the effect size that corresponds to a minimally clinically important difference; (2) inadequate trial selection; and (3) inclusion of different trials of knee and hip OA, spanning from 1 month to 3 years, performed with the prescription product but also with over-the-counter products or dietary supplements, lacking evidence of efficacy or bioequivalence, and resulting in high heterogeneity (I2 = 63%) if these studies had been included in a conventional metaanalysis. Moreover, only trials with 100 patients per arm were considered, but quality of the individual studies was not assessed.
The implications of the predefined effect size of 0.37 as clinically relevant in comparison with lesser effect sizes of well established analgesic medications (e.g., paracetamol, nonsteroidal antiinflammatory drugs) included in international recommendations on hip and knee OA11,12 have been unanimously criticized6,7,9,10. An editorial regarding the use with discernment of OARSI recommendations on knee and hip OA13 emphasized the importance of the treatment effect of slow-acting drugs for OA when considered comparatively with the very strong placebo effect. Although the effect size of these drugs is only slightly larger than placebo, compared to no treatment, placebo induces substantial pain relief (with an effect size of 0.5)14.
In addition, the methods used to analyze studies with multiple treatment groups are unclear. Network metaanalysis (multiple-treatment metaanalysis or mixed-treatment comparisons) allows synthesizing the effectiveness of multiple treatments based on clinical trials that provide direct comparisons of the same but not all of them together. This complex and new statistical technique relies on a strong assumption that studies of different comparisons are similar in all ways other than the interventions being compared, a fact that was. Expert statistical support, as well as subject expertise, is required for a network metaanalysis.
Limitations of the metaanalysis from Wandel and colleagues5, with regard to preventing important methodological flaws and interpretation bias, make the conclusions unjustifiable, in particular the implications for practice, and are in disagreement with all international and European guidelines, which state that chondroitin and glucosamine sulfate may provide symptomatic effects in patients with knee OA and grant the highest level of evidence, 1A, and strength of the recommendation, A, to both products10,11,12. The pitfalls of this metaanalysis were partly addressed in the report from the British Medical Journal post-publication review meeting15, in which it was decided that data from the study did not directly support the strong negative conclusion, so that it is inappropriate to conclude that pharmaceutically produced chondroitin and glucosamine should be discouraged for treating OA.
Conclusions and interpretations from metaanalysis should be scrutinized as critically as those from any other type of study and subjected to reassessment if deemed necessary. Although metaanalyses provide useful information, guidance, and support to clinicians making treatment decisions, they are imperfect and must therefore be used as ancillary to clinical reasoning on a case by case basis13.
Well conducted systematic reviews and metaanalyses are essential tools in the development of clinical practice guidelines and recommendations16. However, recommendations may need to be adjusted to local practice patterns, regulations, and reimbursement policies. Outcomes such as costs/benefits or risks/benefits should also be considered before coming to conclusions on the relevance or not of a therapeutic modality. However, scientific data often based on measurements and statements of statistical probability, although rigorous and valid, should not replace clinical judgement and experience. On the contrary, scientific information should serve as a tool in decision-making for the benefit of optimal patient care.
Footnotes
-
Dr. Henrotin has received education grants from BioIberica, BioXtract, Laboratoires Expanscience, Nestle, Royal Canin, and Artialis SA.
REFERENCES
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.