In their own words, Cochrane systematic reviews are “the benchmark for high-quality information about the effectiveness of health care.” However, Cochrane has been criticised over its lack of transparency in the past, and Hilda Bastian, one of its founding members, thinks there is room for improvement in minimising bias.

The protocol is a crucial component of a systematic review, and is believed to be the key to minimising bias in the results. However, Bastian argues that there is potential for bias in the protocols themselves. For example, there is a section called ‘how the intervention might work’ in the background for Cochrane protocols, which is intrinsically biased by assuming the intervention could work, without actively considering if it could do harm. And even if a perfectly unbiased protocol did exist, if the results from the study contradict the preconceived beliefs of its readers (especially around a controversial topic), then the integrity of the protocol is likely to be criticised anyway.

Bastian proposes that researchers should engage with these potential controversies and be open about the process. Taking inspiration from various systematic review bodies, such as the Agency for Healthcare Research and Quality (AHRQ) and the Cochrane Handbook, her proposed solutions include:

• justifying methodological and analytical choices within the protocol itself, rather than in the discussion (or not at all)
• planning sensitivity analyses on contested aspects
• keeping resulting data open and detailed
• ensuring that every stage of the systematic review process has involvement and oversight from impartial people with methodological expertise.

Focusing on the last step, Bastian is in the process of establishing an Independent Advisory Group for an ongoing update to a Cochrane review. She sees this as a commitment to genuine accountability and responsiveness, and a mechanism for public and open discussion of the protocol. She believes that these steps are critical to ensure ongoing confidence in the integrity of the systematic review.

——————————————————–

Summary by Robyn Foster PhD from Aspire Scientific

——————————————————–

With thanks to our sponsors, Aspire Scientific Ltd and NetworkPharma Ltd