What is peer review?

There is more to science than just conducting experiments. Scientists also spend a lot of time applying for money from funders, reporting their results in research journals, and presenting their work at conferences.

Another important task every researcher does is peer review. Peer review is where scientists check other scientists’ work to ensure only good scientific ideas and results are published and funded.

‘Peer review is a community-led activity and many consider it as an integral part of their job as a scientist,’ says May Copsey, the editor of Chemical Science, a journal that publishes cutting-edge research in chemistry. It’s the job of editors, like May and her team, to ensure papers are peer-reviewed before they get published in the journal.

For each paper submitted to the journal, they find suitable scientists with the right expertise to judge it and recommend whether it should be published. ‘You trust reviewers are giving their best opinion, because they want the best research to be published,’ says May. ‘However, it is an opinion, and these can vary. That’s why we don’t ask one person, we ask at least two.’

Usually, for journals in chemistry fields, the editor keeps the reviewers’ identities secret from the paper authors. This is known as ‘single blind’ peer review. ‘If the authors knew who the referees were, the referees might be afraid to be honest about the paper. This is particularly the case if the authors are well-known and influential,’ says May.

Journal peer review isn’t just about barring poor or incorrect papers from publication. For work that’s almost good enough to be published, reviewers suggest improvements to authors, like ways they could strengthen their arguments with more experiments, or ways they could write their paper more clearly for the benefit of readers.

A modern practice?

The process for deciding what is published in research journals looked very different in the past. For example, from its beginning in 1665, a single editor decided what was published in the world’s oldest scientific journal, Philosophical Transactions of the Royal Society. Later though, the Royal Society was worried that one person could not know enough about the whole range of sciences, so decided in 1752 to replace the role of editor with an editorial committee.

It wasn’t until the 1830s that something like modern peer review emerged. ‘That’s when refereeing comes in’ says Aileen Fyfe, professor of history at the University of St Andrews. The committee’s range of expertise still wasn’t enough to cover all areas of science by that time, so it began formally asking other fellows of the society – referees – to read papers and report on them with recommendations about whether they were any good. But this process still wasn’t called ‘peer review’ until much later, in the 1970s.

Why is it important?

Surely if scientists spent less time reading and writing, they would have had more time to experiment, and advance scientific knowledge? But as Karen Faulds, a professor of analytical chemistry at the University of Strathclyde, UK, points out, scientists need to publish and read others’ work so they can develop ideas about what to research next, and avoid wasting time repeating what others have already done.

‘I always go to peer reviewed literature when I’m researching a topic’, says Karen. ‘If you stick to published work that has been peer reviewed, you know it has been approved by the community.’

Peer review for journals also narrows down the number of things scientists need to search through. ‘If there were no journals selecting papers to publish, and everyone just put all the science they’d done on the internet, it would be really difficult to find the relevant things,’ says May.

Peer pressure

While peer review helps keep scientists accountable, it’s not a perfect system.

Importantly, peer reviewers for chemistry papers are not expected to redo all the authors’ experiments. ‘It really isn’t practical in science,’ explains May. ‘Reviewers don’t have the time to repeat the experiments and they won’t necessarily have the same level of detailed expertise, equipment or right reagents as the people who have done that work.’ They have to trust the author hasn’t made a mistake with the data, and this can be where the peer review process falls down.

‘It can come down to the simplest thing like the scientist has taken the wrong bottle off the shelf and put it into the reaction,’ says May. ‘It does happen.’

There could also be more dishonest reasons for incorrect data. Scientists may be under pressure to find a good result – maybe to increase their chances of promotion – and may be tempted to remove inconvenient outliers, or even fabricate data. It might be hard for a reviewer to spot this, unless they become suspicious the results appear a bit too good to be true.

Peering into the future

People are experimenting with different review models to try to minimise unfairness. ‘Double blind’ peer review – where the authors’ identities are also kept secret from the reviewers – could help guard against reviewers’ personal biases. However, in specialist fields it’s often easy for reviewers to work out who the authors are.

The opposite approach is open peer review. The identity of authors and reviewers are known to each other and to readers: review reports are published alongside the paper. This can encourage reviewers to do a thorough job as they have to publicly defend their opinions. But on the other hand, reviewers can find it harder to be critical when the authors know who they are.

Another variation is where reviewers aren’t chosen or invited: a system where anyone at all can post a review online. This model hasn’t yet been used much in chemistry.

May thinks the single blind system will persist in chemistry publications for the foreseeable future. ‘It’s partly down to the fact people like what they know and are uncomfortable with what they don’t.’

Article by Rowan Frame. Teaching resources by Jenny Koenig, an education consultant based in Cambridge. Jenny has previously taught secondary and tertiary science and has also experienced the peer review process as a biomedical researcher.