A meta-analysis systematically assesses multiple scientific studies to derive conclusions on a specific research topic that may not be clear from individual studies alone. Meta-analyses are used to drive evidence-based decisions in medicine and are often considered to top the hierarchy of evidence for biomedical research. However, a recent article by Jop de Vrieze published in Science questions whether we should always trust the outcomes of these studies.

Using the example of conflicting analyses reporting on the link between media violence and aggression, de Vrieze explores how meta-analyses may produce differing results due to the “many researcher degrees of freedom”. For example, authors may opt to exclude certain study types or apply strict or loose quality criteria to their analyses. Furthermore, de Vrieze considers that author choices may be influenced by biases including financial or ‘intellectual conflicts of interest’, such as a preference to confirm what their own studies have previously shown or to align with certain policies.

As the number of meta-analyses grows, from less than 1000 published in 2000 to 11,000 published in 2017, it may be time to think more carefully about how we conduct and interpret these studies. As described by de Vrieze, some believe that ‘scientists shouldn’t be involved in meta-analyses that include their own work’, while others note that ‘finding suitable authors for a meta-analysis is hard when the people with the most expertise in an area are excluded.’ In controversial cases it may be helpful for research groups with opposing views to set up new analyses together. Overall, transparency is of the utmost importance, so that other researchers can fully understand how a particular outcome has been reached and duplicate the analysis if they wish. In this regard, guidelines such as PRISMA, which aim to improve the reporting of systematic reviews and meta-analyses, are key. Others argue it’s time to go a step further and publish protocols before analyses are conducted and the results known.

——————————————————–

Summary by Louise Niven, DPhil from Aspire Scientific

——————————————————–

With thanks to our sponsors, Aspire Scientific Ltd and NetworkPharma Ltd