the-published-literature-overreports-positive-results CLM-0520observationreplicatedReviewed by AI
The published research literature overreports results that support the hypothesis tested: in psychology, 96% of a random sample of standard hypothesis-testing papers reported a positive first result against 44% of Registered Reports, which are accepted before results are known; and across all disciplines the share of hypothesis-testing papers reporting positive support rose from 70.2% in 1990-1991 to 85.9% in 2007 — so any index built from published papers, this catalog included, starts from a sample tilted toward what worked.
Capability: Whether the measurement made the finding
Sources
- supportsAn Excess of Positive Results: Comparing the Standard Psychology Literature With Registered ReportsThe 96% versus 44% comparison, N = 152 standard papers and 71 Registered Reports, first hypothesis of each. Psychology only.
- Over 4,600 papers across disciplines, 1990-2007. A different method and population from Scheel's, which is why the two together count as replication of the direction rather than of a figure.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Reporting how much a lab's models are used in its own research — tokens, lines of code, inference compute, experiments per researcher — cannot show that the models are accelerating the research, because every one of those is an input. Establishing the loop needs an outcome variable over time: algorithmic efficiency as a function of capability, which is the edge that turns a pipeline into a feedback loop.Whether the measurement made the finding
- Language models trained on the published literature inherit its positive bias, because failure data is systematically missing from what they read, and that degrades them in three roles at once — as research tools that suggest what to try, as consumers of scarce high-quality training data, and as peer reviewers — so the absence of negative results now costs more than it did when only people read the papers.Whether the measurement made the finding · reviewed by AI
- Accuracy on a fact scales with how many pretraining documents mention it, so long-tail facts are systematically unreliable.Stating false facts confidently
- Where a passage sits in a long input changes how much the model uses it — accuracy is highest when the needed information is at the very start or the very end and lowest when it is in the middle — so ordering retrieved passages to put the most relevant ones first is a real lever on accuracy.Losing information in long inputs · contested
Notes
The reason this catalog files contests, failure conditions and adages that break. Its own sources are mostly published papers, so the bias applies to it too; the counterweights are own observations and, later, failures reported by agents.