The idea that we need journals and peer review as ‘proxies for good science work’ because otherwise the academic hiring process would be too difficult is widespread (at least among senior scientists), but when you look at it carefully it has some really troubling implications.

Let’s start from the point of view of an institution that wants to hire someone based on a very granular and detailed analysis of their scientific output. Who should bear the cost of that? Asked in this way, the answer is obvious: the institution itself should bear that cost. In the current system though, the cost is largely paid by everyone other than the institution: in millions of hours of labour doing peer review and paper revisions; in the price of journal subscriptions and APCs that otherwise have no value; and by society as a whole in the opportunity costs in losing out on research that wasn’t done because it wouldn’t lead to a high impact paper, or because the money got spent on a journal subscription. Now all of this wouldn’t necessarily be a problem if the cost was evenly shared by everyone who benefits, and if this way of organising the work didn’t have any negative side effects. Unfortunately, neither of these is true.

The primary beneficiary of the system is the cost saving for the most selective institutions who get the most applications that need to be sorted. In other words, the primary benefits go to the richest, most elite institutions, but the costs are paid equally by everyone.

And we know that the system does have negative side effects. Firstly, there’s the cost to society as a whole in reduction of research output. Secondly, we know that this system is riddled with biases that exclude certain groups of people, which both negatively affects the people excluded and society as a whole in terms of missed opportunities.

So why does such a problematic way of doing things persist? Well partly it’s just a lack of imagination and resistance to change from universities and scientists. Partly it’s because the most influential voices have the most to gain and the least to lose from this system (senior scientists at rich institutions). However, there’s one more troubling aspect. My hypothesis is that the system, as a whole, has discrimination baked in at a fundamental level, and that this discrimination is both financially useful to potential employers, and resistant to criticism because the discrimination is implicit and plausibly deniable.

Publication decisions are partly based on the scientific qualities of the paper, but also partly based on non-scientific aspects of the authors’ ability to navigate the system, which can be influenced by things like who they know, whether they went to the right school, etc. Universities are happy to select based on that non-scientific ability because it’s predictive of future funding success (for example). However, those non-scientific aspects are highly correlated with protected characteristics like race, ethnicity, gender and socioeconomic status. Given that funding success is a goal, and that protected characteristics are predictive of this success, universities have a financial incentive to engage in discrimination based on these characteristics. However, they can’t do so explicitly because it’s illegal. Journal publication record provides cover, both because it’s external to the university, and because the decisions are very noisy, making it hard to establish in any single case that the publication decisions were discriminatory.

Individual university managers and scientists involved in hiring decisions don’t have to be explicitly discriminatory for this global effect to take place, indeed they are likely not even aware of the effect in many cases. However, since attempts to deviate from this practice will likely have measurable financial consequences, and the university system is extremely competitive, we can see how it becomes hard to change, even if the majority of individuals involved would be horrified to think of themselves as engaging in such a large scale discriminatory process.

In summary, I think we should resist the argument that we have to keep using journal publication record as a key indicator of applicant suitability, because it’s a system that shifts costs from the rich to the poor, slows down scientific progress as a whole, and bakes in discrimination at a fundamental level.

This post came out of a post in a discussion on Mastodon, available here.