Published methods do not contain what it takes to repeat the work. The missing part is in someone's hands.
The reproducibility problem is usually described as a statistics problem. A large part of it is a documentation problem: the judgment calls, timings and "you can tell when" steps that never reach the methods section.
Experiments from 53 high-impact cancer papers that could be attempted without asking the original authors for clarification. A third of authors did not help or never replied; 68% shared no data at all. Only 26% of the planned experiments could be completed.
Of 1,576 surveyed researchers had tried and failed to reproduce another scientist's experiment. More than half had failed to reproduce their own. 52% said there is a significant crisis.
Estimated US spend on preclinical research that cannot be reproduced.
The methods section was never the whole method
A methods section is written from memory, after the fact, under a word limit, for a reader who is assumed to know the field. The steps that make the experiment work — how long "until it looks right" is, what a loose pellet looks like, which lot of antibody was the good one — are exactly the ones that get left out, because to the author they are obvious.
The knowledge lives in one person
When that person leaves the lab, the protocol leaves with them. When they train a replacement, something is lost in the teaching. The reproducibility literature measures the cost of that loss at the scale of a field; every lab feels it at the scale of a failed run.
The analysis is the least documented part
The methods section describes the bench in detail and the analysis in a sentence: "expression was normalized to GAPDH and plotted." The export, the spreadsheet, the replicate that was dropped, the threshold set by eye, the software version — none of it is written anywhere, and it is exactly where two people analysing the same data get different answers. Many scientists run these steps by hand without knowing an automated tool already exists, because no one has ever asked them to name the step.
What a complete protocol has to state
- Materials with identifiers: supplier, catalogue number, lot, storage, preparation, what "fresh" means.
- Quantities with tolerances: volumes, concentrations, temperatures, times, speeds, and the range that still works.
- Order and timing: what must happen immediately, what can wait, where the pauses are and how long they can be.
- Judgment points with rules: what you are looking at, the decision rule, what a wrong call looks like, how you recover.
- Controls and checks: what runs alongside, what result would stop the run.
- After the wet lab: what is done to the data, by hand and by software, with settings and thresholds.
Funders are turning this list into a condition of the money. See who it is for.