Both positive and negative results must be made available to allow scientific advancements to be fully understood. However, it is estimated that for clinical trials alone, positive findings are nearly twice as likely to be published as negative or inconclusive outcomes, creating a publication bias. Sandra Petty is an academic and clinical neurologist focusing on epilepsy and its comorbidities, and is founder and CEO of the Center for Biomedical Research Transparency (CBMRT). Following her talk at the 2020 European Medical Writers Association Symposium, we found out more about the CBMRT’s efforts to reduce publication bias through increasing the reporting of negative results.

Firstly, for anyone who is not already familiar with the CBMRT, please would you describe the organisation and its aims?

“The CBMRT is a 501(c)(3) not-for-profit organisation that facilitates transparent reporting of biomedical and clinical research. We aim to ensure that all biomedical and clinical research results – including negative and inconclusive results – are accessible in the interests of patient safety and research efficiency.”

We aim to ensure that all biomedical and clinical research results – including negative and inconclusive results – are accessible in the interests of patient safety and research efficiency.

What inspired you to found the CBMRT?

“Publication bias has been a major issue affecting biomedical and clinical research for a long time. Working both clinically and in research, I could see major issues for research efficiency and patient safety that are related to non-publication of ‘negative’ results. The problem is that many results of well-performed research never see the light of day – particularly when an expected effect is not observed.

The problem is that many results of well-performed research never see the light of day – particularly when an expected effect is not observed.

Negative, inconclusive and replicative results compete against new studies with positive results for limited publication space in the high impact journals. There’s sometimes a perception that publishing negative results may harm career prospects, and/or a view that investing the time to write up the result would yield a low impact factor publication, so time may be better spent focusing on higher impact publication areas. All of this works to the detriment of gaining a complete and balanced understanding of the area of research.

The enormous advances made in our understanding of diseases and therapies to date are a direct result of the quality of research conducted by dedicated scientists. Their work inspires, informs and refines new avenues of enquiry. The ultimate beneficiaries though are patients – clinicians are better placed to optimise treatments and to manage risks. So, the stakes are high if we fail to achieve balanced and transparent reporting of well-performed research (regardless of the results): it leaves us with an incomplete understanding of the state of our field and of our treatments, and it affects the knowledge we share with research participants and patients. And we’re wasting taxpayer and donor money by not sharing the results of well-executed research that they’ve funded.

The stakes are high if we fail to achieve balanced and transparent reporting of well-performed research (regardless of the results): it leaves us with an incomplete understanding of the state of our field and of our treatments, and it affects the knowledge we share with research participants and patients.

So, to address these issues, I co-founded the CBMRT after completing my postdoc. The other motivation in founding the CBMRT was to bring stakeholders from across the biomedical and clinical research transparency ecosystem closer together through annual convenings to facilitate updates on important issues and developments in the field.”

You have previously discussed the causes and consequences of ‘dark data’ in scientific research. Can you outline why it’s so important to publish negative data, and what’s holding researchers back?

“Most scientists will tell you that there is ‘dark data’ sitting in lab books around the world. While we can’t necessarily publish every small experiment, when research is well-performed, a more complete record of what has been done, and what works and doesn’t work, is a really useful tool. The reasons behind dark data are more complex, ranging from opportunity costs of writing up papers to the perception that even if written up, negative/null results might be less likely to be published, or may reflect negatively on a researcher. At its worst, dark data can also include outright fraud or deliberate hiding of results, which is the more extreme end of the spectrum. Institutions often reward high impact factor articles in terms of key performance indicators and tenure conditions, which may lead scientists under time pressure to prioritise their positive findings for publication, sometimes at the expense of other results.

The reasons behind dark data are more complex, ranging from opportunity costs of writing up papers, the perception that even if written up, negative/null results might be less likely to be published, or may reflect negatively on a researcher.

The problem with dark data is that scientists and clinicians cannot review it. They can’t see that a preclinical study showed no effect, or perhaps no effect and a side effect, leading to problems with research translation that might have been addressed or avoided if the publication record had been more complete.

A paper is still one of the gold standard publication methods, allowing authors to more fully explain their work than by simply sharing data or results; it creates context, and a reference point which should be available for systematic review and consideration before new projects (and funding) are commenced.

A paper is still one of the gold standard publication methods, allowing authors to more fully explain their work than by simply sharing data or results.

Addressing this issue and balancing the publication record is ultimately useful for scientists, clinicians and maintaining public trust in the scientific process.”

The CBMRT’s Null Hypothesis Initiative promotes the publication of negative, inconclusive or replicative results, so far partnering with Neurology, the American Heart Association (AHA) journals and Neurotrauma Reports. Do you think it’s important that negative data are published in mainstream journals, rather than venues dedicated to null outcomes?

“When I co-founded the CBMRT, the Null Hypothesis Initiative was my central concept, creating dedicated space for well-performed negative, inconclusive or replicative studies to be published in pre-existing journals – rather than in new standalone journals. Working with established journals allows us to keep arms-length from the review process and remain independent, while also working to promote an awareness of, and a solution to, publication bias. At the same time, we promote the publication of these articles with editors, editorial boards, publishers and authors, aiming to enhance publication culture and reduce bias.

The Null Hypothesis Initiative was my central concept, creating dedicated space for well-performed negative, inconclusive or replicative studies to be published in pre-existing journals – rather than in new standalone journals.

I do believe it is important to publish well-performed studies in mainstream journals, regardless of the results. This should really be a standard part of scientist and journal workflow. These journals are regularly circulated to readers who are usually clinicians and scientists in the field and are also the journals where they aspire to publish articles. This then adds to awareness of the issue in question and drives further paper submissions in the target areas.”

How has the initiative been received so far, and what are your plans for its future?

“Anecdotally, we have received a lot of positive feedback from the research and transparency community, as well as from authors who have found the Null Hypothesis Initiative really helpful in publishing their work. Neurology and the American Academy of Neurology were our original collaborators for this initiative, which has been very successful. We are proud to be working with the AHA’s journals and with Neurotrauma Reports (and Cohen Veterans Bioscience) to establish Null Hypothesis papers in these fields.

We aim to have the Null Hypothesis Initiative evolve as a grass-roots scientist-driven movement across as many fields as needed, and we’re looking at fields and funding sources to help achieve this. Ideally, I’d like to see all of the major funders and publishing houses come onboard to sponsor and support this initiative.”

I’d like to see all of the major funders and publishing houses come onboard to sponsor and support this initiative.

Null Hypothesis articles are made freely available. How important is the open science movement for improving transparency in research?

“We fundraise and collaborate with publishers and funders to support free-to-read and open access publications, so that the information is immediately available to clinicians and researchers. I believe that the open science movement has great potential to enhance transparency in biomedical and clinical research through available frameworks for recording and sharing protocols, data and results. However, researchers need to be supported to use these tools through grant structures and academic benchmarks to reflect their contributions.”

Alongside the Null Hypothesis Initiative, the CBMRT hosts the Biomedical Transparency Summit (BMTS) series. This year, the series will be held as a free virtual event for the first time. What can attendees look forward to?

“The BMTS is one of the real highlights of our year! The previous format was a full day Summit with fabulous speakers, important updates, networking and discussion among stakeholders in transparency, held in the US and EU. This year, we have shaken things up and are holding our Summit series online as three separate one-hour webinar sessions over February and March.

Again, we’ll be focusing on the latest developments around the world in the biomedical research transparency space. Just like our in-person Summits, participants will be updated by engaging experts who are leading transparency efforts in the policy, industry, technology, academia, publishing and funding domains. We’re delighted yet again by the quality (and enthusiasm) of our amazing speakers. Our goal is for the webinars to be as interactive and stimulating as possible so presentations will be deliberately brief (but content rich) to allow ample time for productive discussions amongst participants and speakers.

Here are the details:

• Webinar 1: Research integrity – developments across the Atlantic [note: read our summary of the webinar here]. Keynote speakers: Prof Ana Marušić (Standard Operating Procedures for Research Integrity [SoPs4RI] Project Leader) and Michael Lauer MD (Deputy Director for Extramural Research at the National Institutes of Health [NIH])
• Webinar 2: Open access – developments across the globe [note: read our summary of the webinar here]. Keynote speakers: Prof Johan Rooryck (Executive Director, cOAlition S) and Dr Ginny Barbour (Executive Director, Australasian Open Access Strategy Group)
• Webinar 3: Acceleration of research and implications for research transparency [note: read our summary of the webinar here]. Keynote speakers include: Prof Ida Sim (Director, University of California San Francisco [UCSF] Informatics and Research Innovation), Dr John Inglis (Executive Director, Cold Spring Harbor Laboratory Press), Deborah Dixon (Global Editorial Director, Oxford University Press)

Registration is free. As we are a non-profit organisation, donations are always very much appreciated. Full details, including speaker bios, can be found in the Summit series outline and on the flier, and you can register here.”

Finally, looking ahead, what more can be done to support publication of high-quality research, regardless of the outcome, and what are your top tips for researchers with negative or null results?

“Let’s stop and think why ‘negative’ results of well-performed studies attract negative connotations. We need to reframe the thinking and perception here – they are valid results of well-performed research.

Let’s stop and think why ‘negative’ results of well-performed studies attract negative connotations. We need to reframe the thinking and perception here – they are valid results of well-performed research.

If one strategy has not solved the problem it is important to publish that finding so that research waste is reduced, and other methods or approaches can be considered. The clinical research problem itself isn’t solved by a null finding but it may well lead towards something that does work.

To support publication of high-quality research regardless of the outcome, we need to address cultural and structural barriers, make space in journals and celebrate a culture of transparency as something that enables science to progress more efficiently.

My top tips for negative or null findings:

• Write them up and submit them.
• Point out their quality and value in your cover letter. We all need to create positive press for ‘negative’ data. It’s your hard work, and acknowledges contributions of study participants, collaborators and funders.
• If you don’t think that a full-length paper is appropriate, look at submitting a preprint, sharing the data or using open science tools to share your work.
• If you think your field is in need of a Null Hypothesis partnership, please get in touch!”

Associate Professor Sandra Petty is an academic and clinical neurologist at the University of Melbourne Medical School, St Vincent’s Hospital and Alfred Health in Melbourne. She is founder and CEO of the CBMRT. You can get in touch with her at sandy@cbmrt.org and follow the CBMRT on Twitter @CBMRT_org.

——————————————————–

With thanks to our sponsor, Aspire Scientific Ltd

---