Defending science in public we often talk about ‘peer reviewed science’. But could this framing contribute to undermining trust in science and holding us back from improving the scientific process? How about instead we talk about the work that has received the most thorough and transparent scrutiny?

Thorough scrutiny is the main rationale for peer review (although its historical origins are a bit murkier). The idea is that we should have more trust in science that has been scrutinised by independent scientists. This rationale is entirely correct, but traditional peer review is a blunt instrument to achieve this. Firstly, it is limited only to a small number of reviewers (often just 1-2). Secondly, the reviewers are often not the most qualified scientists to scrutinise the work. Instead, they’re the people who agreed to do it, often after many others were asked. Thirdly, due to academic time pressures, the majority of peer reviewers only spend a small amount of time doing reviews. Given all this, it is not surprising that peer review fails to catch the majority of errors in papers. Finally, since there is nothing to stop you resubmitting a paper to another journal, and there are many journals out there who just want to get the publication fees, eventually every paper will get published if the authors are sufficiently motivated.

Traditional peer review therefore lets a lot of bad papers get published. Since the process is not transparent (most journals don’t publish reviews), it can be hard to distinguish between a paper that provides solid results and has been thoroughly tested by the right peer reviewers, and something that has received little to no real scrutiny. Journal reputation gives us a little more information, but not as much as we think. Papers in journals with the strongest reputations are more likely to be retracted, for example.

Even if journal reputation did provide a useful signal as to the degree of thoroughness of the scrutiny received by a paper, this isn’t knowledge easily accessible to the general public. There is a danger in this. If we insist - in our scientific communications - that the general public only look at “peer reviewed science” (a common theme), then we are inviting them to make no distinction between papers that have been published at a predatory journal after a string of rejections, and papers that have passed a tough test from the right peer reviewers. If these poor quality papers (that may well be the large majority at this point) get shown to be wrong, perhaps dramatically so, that then reflects badly on all of science and undermines trust in science.

We need to abandon the idea that peer review is a ‘gold standard’ because it never was and this idea harms science and public trust in science. We need to go back to the stated rationale behind peer review: that scientific work should receive thorough scrutiny. We should insist that this scrutiny be transparent. We should take steps to create a new scientific culture where the more transparently and thoroughly scrutinised work is given higher attention and status. And, we need to educate the public that this is what makes science work.

Moving towards a culture that places a much higher value on transparent and thorough scrutiny has a number of other positive effects.

Firstly, it lets us experiment with new models of how that scrutiny should be organised. If we insist on traditional peer review, there’s no incentive to try other approaches and those experiments are actually punished, precisely because they differ from the traditional approach. However, if we have a common standard (transparent, thorough scrutiny) instead of a common process (traditional peer review), we can experiment with different ways of achieving that. This will be sorely needed in the near future, as traditional peer review comes under increasing strain both from the volume of papers being written and the rise of large language models.

Secondly, it would allow us a much clearer measure of which scientific institutions were contributing to good scientific processes, and make it easier to criticise those that don’t. For example, ‘predatory journals’ that do minimal peer review and only want to collect publication fees will either have to open up their processes and show just how little they value thorough scrutiny, or remain closed and show they don’t value transparency.

Finally, it gives authors a mechanism to increase the value of their research beyond getting it published in the right place. By sticking their neck out and subjecting themselves to increased scrutiny, they increase the value of their work if it survives this more rigorous process. They can do this by allowing their work to be openly reviewed in an ongoing way. They can make their raw data and code open. They can even go really far beyond even the most rigorous of proposed standards, and invite scientists who have doubts about their results to come and visit their lab and see how the data was collected, and then publish their own reflections on what they saw. These are all signals that the authors believe in their results. If others don’t find major problems despite everything being open to inspection, that should hugely increase our trust in the results and reflect well on the authors.

This is what would happen to science if we moved towards a culture that explicitly recognised the value of thorough and transparent scrutiny.

It will be difficult to achieve this, certainly. Traditional peer review is a known quantity, has worked well for many people, and has strong institutional backing. Change is always hard. However, there is a path forwards here. We don’t have to immediately abandon everything we’re doing now and create whole new systems. We just need to start building a culture that values transparent and thorough scrutiny. We can start this by talking about it with students. We can discuss the extent to which a paper opens itself to it when we write formal reviews or when we talk about it informally in conversations (in person or on social media). We can bring up this concept when we talk to journalists and the general public. We can use this concept when we decide on the reputation of a journal, or evaluate a candidate for a job or grant.

Once the idea has built a certain momentum in the informal culture of science, only then do we need to think about whether we want to formalise it, and if so how. Indeed, we might realise that this step simply isn’t necessary once we get to that point.

So get out there and start talking about science like this and let’s make science better and more trusted.

(And if anyone has a catchier phrase than thorough and transparent scrutiny please tell me, I know it’s not the most inspiring. 😉)