Artificial intelligence (AI) is quickly becoming integrated into our daily lives, whether through virtual assistants on mobile devices, online shopping algorithms, or safety mechanisms in cars. It has been suggested that AI could be used to address lengthy peer review times, with some research indicating that these could be shortened by 30% through the use of algorithms. A recent article in Humanities and Social Sciences Communications considers the potential benefits and pitfalls of increasing use of AI in peer review.

Dr Alessandro Checco and co-authors designed an AI tool and trained it on 3,300 peer reviewed conference proceedings where the reviewer’s recommendations were known. They then tested the tool to assess its ability to determine whether papers would be accepted and how they might be scored by reviewers. They found that the AI tool was often successful in predicting peer review outcomes, demonstrating strong correlations between peer review scores and superficial characteristics such as readability and formatting.

The AI tool was often successful in predicting peer review outcomes, demonstrating strong correlations between peer review scores and superficial characteristics such as readability and formatting.

The authors suggest that, if such proxy measures are typically indicative of the overall quality of a paper, the AI tool could successfully reduce the burden on peer reviewers by performing initial screening checks (and ‘desk rejecting’ as needed). However, there is also a risk that the AI tool itself could replicate the biases that human peer reviewers are susceptible to. For instance, language patterns in papers written by non-native English speakers may receive low scores from an AI tool regardless of research quality, reducing the likelihood of acceptance for publication. This type of systematic bias could disproportionately impact authors from lower- and middle-income countries, who may already face financial barriers to getting their research published. On the other hand, the authors are optimistic that such AI tools could make it easier for peer reviewers to spot their own biases, providing them with an opportunity to make their reviews more equitable.

AI tools could make it easier for peer reviewers to spot their own biases.

Given the ever increasing number of manuscript submissions, there is a need to critically assess the traditional peer review process and identify ways it can be streamlined. Peer review is a central mechanism in maintaining high quality research and outcomes can play a role in career progression. While the authors are not looking to a future where AI tools completely replace human peer reviewers, they could be a promising way to address the evolving needs in scientific publishing. However, further research is needed to carefully consider any ethical pitfalls.

——————————————————–