KEY TAKEAWAYS

• Journal impact factor is a poor predictor of the quality of peer review.
• Implementing open peer review and reviewer training would facilitate the assessment and quality of peer review.

Journal impact factor is often considered an indicator of quality, but does a high impact factor necessarily equate to high quality peer review? In a recent Nature News Q&A article, Dr Anna Severin discussed how her team used artificial intelligence to investigate whether there is any link between the two. Their results were also reported in a recent preprint on arXiv.

As Dr Severin explained, ‘quality’ peer review is difficult to define. Authors want suggestions for improvement, while editors want recommendations on whether to publish. As a result, Dr Severin and her team developed the following proxy categories for quality:

• Thoroughness: comments made on the materials and methods, presentation and reporting, results and discussion, or the importance of the article.
• Helpfulness: comments relating to praise or criticism of the article, giving suggestions for improvement, or providing further examples.

The team randomly selected 10,000 peer review reports submitted to more than 1,600 medical or life science journals. A random sample of 2,000 sentences were then coded to the above categories. The team used this information to train machine learning models to predict categories for over 187,000 sentences across the reports.

Overall, peer reviewers from higher impact factor journals focused on methodology, but spent less time suggesting improvements and providing examples than reviewers from lower impact factor journals.

Reports in higher impact factor journals tended to be longer, and reviewers were more likely to be from Europe or North America. Although there seemed to be a link between impact factor and peer review quality, any differences were modest, and variability was high. As such, the team concluded that impact factor is “a bad predictor for the quality of review of an individual manuscript”.

The team concluded that impact factor is “a bad predictor for the quality of review of an individual manuscript”.

Dr Severin recognised the limitations of using artificial intelligence, but considered this to be a first step in assessing peer review quality in a systematic and scalable way, and called for the following actions:

• Reviewers should be trained and given clear guidelines.
• Peer review should be open rather than confidential.

—————————————————–