On the "reproducibility crisis" in science

Posted at 10:00 on 08 January 2020

I've had two or three people tell me about the "reproducibility crisis" in science in the past few months. The most recent such comment was at the weekend, which coincidentally came right at the time when a 2016 Nature article on the subject was at the top of Hacker News. Here are some thoughts on the matter.

First of all, I'd like to make it clear that the reproducibility crisis doesn't call the entire scientific method into question right across the board. There may be a lot of papers published in the scientific literature that can't be replicated, but there are also vast swathes of others that can and are -- often by multiple independent methods. The fact that some studies can't be reproduced says nothing whatsoever about the validity of the ones that can, and it's the ones that can that go on to establish the scientific consensus and make their way into school and university textbooks.

In fact, it's only to be expected that the scientific literature would contain a sizeable proportion -- perhaps even a majority -- of non-reproducible studies. Scientists are only human, and if they rarely if ever made any mistakes, then that would suggest there was some form of underhanded collusion going on. It's all too easy for them to inadvertently end up making mistakes, taking shortcuts, or writing down lab notes that don't accurately describe exactly what they did. But that is why science demands reproducibility in the first place -- to filter out problems such as these.

It's important to realise that the reproducibility crisis only really affects the very frontiers of science -- cutting edge research where the practices and protocols are often still being developed. There will always be a certain amount of churn in areas such as these. It rarely if ever affects more well established results, and it's not even remotely realistic to expect it to cast any doubt on the core fundamentals. We can be absolutely confident that subjects such as relativity, quantum mechanics, Maxwell's Equations, thermodynamics, the Periodic Table, evolution, radiometric dating, Big Bang cosmology and so on are here to stay.

Furthermore, scientists are actively working on ways to improve things. There is a whole scientific discipline called "meta-science," which is devoted to increasing quality while reducing waste in scientific research. That is why scientists have adopted techniques such as peer review, blind studies, statistical methods to detect fraud (using techniques such as Benford's Law) and the like. One recent innovation has been pre-registration of clinical trials as a means to combat publication bias and selective reporting: in many cases, the studies are peer reviewed before the results are taken rather than after the fact.

Interestingly, the disciplines that are most profoundly affected by the "reproducibility crisis" are the social sciences -- sociology, psychology, medicine, and so on. These are subjects which first and foremost concern the vagaries of humans and other living beings, which deal with very imprecise data sets with wide spreads of results, and which predominantly rely on statistics and correlations that are much more open to interpretation and studies that are qualitative rather than quantitative in nature. It is less of a problem for the more exact sciences, such as physics, chemistry, mathematics, geology, astronomy, or computer science.

The thing about science is that its foundations of testability and rigorous fact-checking tend to bring it into direct conflict with dishonest people, hidden agendas, and vested commercial or political interests. Consequently there is no shortage of people who will do whatever they can to try and undermine public trust in the scientific community and even the scientific method itself. One of the ways that they do so is to take real or perceived imperfections and shortcomings in science, blow them out of all proportion, and make them appear far more significant and far more damaging to the legitimacy of scientific scrutiny than they really are. But that's just dishonest. Science may not be perfect, and non-reproducible papers may be plentiful, but nobody gets a free pass to reject anything and everything about science that they don't like.

Featured image: United States Air Force Academy