Researchers test new Risk of Bias assessment tool for non-randomized intervention studies
The Cochrane Risk of Bias Tool for Non-Randomized Studies of Interventions shows potential value in systematic reviews in an assessment published this week in PLOS Medicine. The study, conducted by researchers at Dalla Lana School of Public Health at the University of Toronto, Ontario, Canada demonstrated that exclusion of studies with higher Risk of Bias (RoB) assessed by a new instrument developed by the international Cochrane Collaboration changed estimates of clinically pertinent outcomes in two published systematic reviews of cardiovascular risks associated with use of two popular classes of drugs.
Inclusion of studies at high RoB may lead to erroneous conclusions about the benefits and harms of medical interventions. Researchers undertaking systematic review and meta-analysis (SRMA) of randomized clinical trials usually measure RoB of the component studies using an established tool, but there is no widely-accepted measure of RoB for non-randomized studies. As a test of performance, the Toronto-based researchers applied the new Cochrane RoB instrument, called ACROBAT-NRSI, to 37 papers included in two widely cited systematic reviews of the use of thiazolidinediones (“glitazones” used to treat diabetes) and cyclooxygenase-2 (COX-2) inhibitors (used to treat inflammation), and risk of major cardiovascular events (myocardial infarction and heart failure) and death.
Assessment took roughly 2.5 hours per study, and inter-rater agreement was moderate to substantial (overall Kappa 0.45 to 1.0). Of the 37 studies originally included in the meta-analyses, only eight had low RoB scored by ACROBAT-NRSI. Confining meta-analysis to studies with low RoB shifted the pooled odds ratio for myocardial infarction for rosiglitazone versus pioglitazone from 1.14 [95% CI 1.07-1.24] to 1.06 [95% CI 0.99-1.13]). Estimates of pooled relative risks (RR) of cardiovascular events with COX-2 inhibitors compared with no nonsteroidal anti-inflammatory drug changed little when analyses were confined to studies with low or moderate RoB, with the exception of an increased RR associated with ibuprofen (from 1.07 (95% CI 0.97-1.18) to 1.14 (95% CI 1.03-1.26)).
Because the study was conducted by a specific two-person team and considered only two systematic reviews, the findings may not be generalizable and further testing of the new Cochrane instrument is needed. Nonetheless, the findings indicate that at least two published meta-analyses included studies with high RoB. The authors state, “[e]xclusion of studies deemed to have unacceptably high RoB may impact the findings of pooled estimates of intervention effects, altering both the statistical and clinical significance of the results.”