Marcus Munafò, Professor of Biological Psychology at the University of Bristol and Chair of the UK Reproducibility Network (UKRN) Steering Committee, studies how reducing bias and increasing reproducibility and replicability can improve the quality of research. Following his recent talk at the 2020 European Medical Writers Association (EMWA) Symposium, we spoke to Munafò about how incentives shape science and the future of reproducibility in research.

For anyone who is not already familiar with the UK Reproducibility Network, could you describe some of the work that the organisation does to ensure that the UK remains a centre for high-quality research?

“I think our main role is to support and help coordinate activity that is already happening. For example, there are a huge number of community-led (and often early career researcher-led) initiatives – ReproducibiliTea, RIOT Science Club and so on – that we’re keen to support and promote in partnership. We also have local networks – self-organising groups of researchers keen to develop and promote effective research practice. Our aim is to encourage and support these networks with activities such as adopting the initiatives I’ve mentioned. We’re working with institutions to coordinate things like training and changes to promotion criteria. We have a wide range of external stakeholders – funders, publishers, learned societies and sectoral organisations that we work with too. This layered structure allows us to help coordinate activity both vertically and horizontally, and also provides us with the opportunity to give our community or researchers (via our local networks) the opportunity to have a voice at other levels.”

Can you explain how reproducibility and replicability are linked to the methodological quality of a study, and how this relates to transparency?

“There are various definitions of reproducibility and replicability, but both are essentially a way that research is made accountable. Being able to reproduce a research workflow and replicate the findings generated by it is one way of ensuring that the findings are robust. Of course, this accountability framework applies to only some disciplines, and one thing we’re keen to do is explore the different frameworks that apply across disciplines (including what different disciplines – from sciences to the humanities – can learn from each other).

One thing we’re keen to do is explore the different frameworks that apply across disciplines (including what different disciplines – from sciences to the humanities – can learn from each other).

Transparency is something that is probably quite a broadly applicable framework that allows other researchers to scrutinise not only the output of a research process, but the intermediate steps that led to it. This, in turn, allows for errors to be detected, or sources of disagreement to be identified and perhaps discussed.”

You have spoken previously about biases, including publication incentives and pressure for prestige and innovation that may negatively influence a scientist’s research and publication decisions. Can you tell us a little more about this, and how such pitfalls might affect the scientific literature in practice?

“Researchers are human, and therefore subject to the same cognitive biases as everyone else – confirmation bias, for example, which is the tendency to interpret (and even seek out) information that confirms one’s prior beliefs or values.

This means that our passion for our subject is a double-edged sword: we might become personally invested in our findings or theories, and (unconsciously) seek out findings that support them, rather than disconfirm them. We also respond to incentives, often unconsciously. The fact that career advancement depends on factors including publication means that we are incentivised to publish, rather than necessarily to get to the truth.

The fact that career advancement depends on factors including publication means that we are incentivised to publish, rather than necessarily to get to the truth.

We all presumably want the latter, but the system we work within implicitly nudges us to focus on the former. This combination of cognitive biases and incentive structures is important to understand if we want to both protect ourselves from the effects of these biases and develop better incentives.”

In your opinion, what role should academic institutions play in tackling the pressures placed upon researchers? Can you see any promising signs of such activities?

“In many ways, institutions are the repository of much of our research culture and shape the incentive structures that we work within – for example, through hiring and promotion criteria and processes. This is why we ask institutions that join UKRN to create senior academic positions focused on research improvement.

This allows us to coordinate not only training activity (to ensure researchers across those institutions ultimately receive common foundational training in core skills), but also to work collectively (and in partnership with researchers themselves, via our local networks) to reflect on and hopefully improve our incentive structures. For example, several of our institutions now include open research practices in their promotion criteria, and have launched open research prizes. If we consider transparency in research to be a positive thing, we need to directly incentivise it.”

If we consider transparency in research to be a positive thing, we need to directly incentivise it.

What can scientific journals do to address challenges with reproducibility and replicability? What role do open access policies play in this?

“Journals (and publishers more generally) can do a lot to go beyond open access and promote open research more generally. Data sharing requirements are becoming stronger for example, and reporting guidelines are also becoming more widely adopted. We recently ran a series of workshops on open research in partnership with the publisher Wiley, with the content of each workshop shaped in part by the host local network. In the spirit of coordination, we’re also piloting initiatives that link journals and funders, such as the Registered Reports Funding Partnership initiative. This links grant and journal peer reviews, so that successful applicants receive not only funding to conduct their research but also in-principle acceptance of the eventual work, before a single datum has been collected.”

A recent report by the European Commission into the reproducibility of scientific results emphasised that steps should be taken at the very start of a research project to ensure its integrity. Do you find that researchers are generally taking these steps, or is there a greater need for this kind of planning?

“I think this is becoming more and more mainstream. Early career researchers are increasingly motivated to adopt open research practices, for example. However, senior researchers are also becoming increasingly aware of the importance of these practices (not least because they are required, or at least encouraged, by funders and journals). This is positive, because it creates incentives – the funder and journal requirements incentivise senior researchers to begin adopting these practices, which in turn means that they need to hire early career researchers with those skills. And that creates an incentive for those researchers.

Another advantage of open research is that it provides opportunities to demonstrate a far broader range of outputs – pre-registered protocols, data deposits, code, materials, preprints and so on. Those who can demonstrate those skills are, I think, highly employable.”

The European Commission report also talks about the importance of public trust in science. Do you think this topic has become more urgent given the COVID-19 pandemic and development of vaccines?

“In general, the public continues to trust science and scientists, and I think rightly so. Although we can improve how we work (and should always be reflecting on how to do so), I don’t think we should be too nihilistic. Academic research continues to provide important insights and solutions to pressing problems. The COVID-19 pandemic has illustrated this, although it has also highlighted some of the problems with current incentive structures. One key distinction – made by Simine Vazire, David Spiegelhalter and others – is that we need to move away from a culture that requires us to trust individual researchers, and to one where the research process is inherently more trustworthy (for example, by being more transparent).”

We need to move away from a culture that requires us to trust individual researchers, and to one where the research process is inherently more trustworthy.

Finally, your 2017 Manifesto for Reproducible Science outlined key stakeholders and steps that can be taken to improve transparency throughout the scientific process. What changes, if any, would an updated manifesto contain given the progress you’ve made in the years since its publication?

“I think for the most part, what we wrote still stands. But one area where I think we could have been stronger was in discussing the role we all have to play in driving change. We originally wrote that researchers themselves could make a difference by adopting open research practices. That is certainly true, but we can all do more. As we review manuscripts or grants, or recruit PhD students or staff, we can enact the principles we care about. We can all be agents of change, regardless of our career stage.

It’s easy to under-estimate how powerful this kind of collective action can be. By creating a framework that helps to coordinate activity (for example, across our local networks and institutional leads), and also provides a voice for grassroots researchers across those levels, UKRN can hopefully help realise this change.”

Marcus Munafò is Professor of Biological Psychology at the University of Bristol and Chair of the UKRN Steering Committee. You can contact Marcus via marcus.munafo@bristol.ac.uk and follow him and the UKRN on Twitter @MarcusMunafo and @UKRepro, respectively. Follow these links to find out more about the UKRN, along with other reproducibility networks in Australia, Germany, Slovakia and Switzerland. 

——————————————————–

With thanks to our sponsor, Aspire Scientific Ltd

---