Scientific conclusions should be reproducible. If A+B = X today, then all things being equal, A+B should = X tomorrow. But if A+B inconveniently = Y, then there was probably something wrong with the original idea that A+B = X and a new hypothesis may be required. As this happens frequently in science, researchers must repeat experiments to ensure initial results aren’t anecdotal flukes.
At the individual level repeating experiments is not a problem. If an individual scientist can run one experiment, they can (time and money permitting) run another. It may be boring, but it is feasible. Experimental reproducibility is required by peer-reviewed journals — and although there is no general threshold of accepted reproducibility, most authors demonstrate the fidelity of their results through the medium of ‘statistical significance’.
Thus at the individual level, experimental reproducibility is rewarded in the currency of research: Publications. So far, so good.
The next tier of experimental reproducibility is the replication of results by other scientists. This is an unrewarding endeavour. In fact, due to the bias of selectively publishing ‘new’ work, there is an active discrimination against publishing repeated results in leading (i.e. career bumping) journals. This anti-replicant discrimination is so fierce that researchers often live in fear of competitors publishing ‘your findings’ before you do. There is no prize for being second.
As publications drive careers, there is no mainstream incentive for researchers to spend their time and money replicating other people’s experiments. As a result, many conclusions, at the publication/community level are n=1.
This highlights a bizarre paradox. On one hand, statistically significant reproducibility is the essence of individual science. Yet, at the community level, reproducing existing results is career suicide.
Tom Bartlett in Percolator reports on a charming challenge to this trend. In short, a group of researchers (lead by the Open Science Framework) have decided to replicate every study from three Psychology journals for a year. Their aim is to “estimate the reproducibility of a sample of studies from the scientific literature”.
I can’t wait to see what they find.