arxiv-2604-06236 SRC-0582 · paper

LLMs Have Made Failure Worth Publishing

Sungmin Lee

Created: 2026-04-04 · Ingested: 2026-10-19

Scientific publishing systematically filters out negative results. We argue that this long-standing asymmetry has become an urgent problem in the era of large language models, which inherit the positive bias of the literature they are trained on, face an impending shortage of high-quality training data, and are increasingly deployed as both research tools and peer reviewers. We analyze three ways in which LLMs have changed the value of failure data and show that the systematic absence of such data degrades their utility as research tools, training data consumers, and peer reviewers alike. We outline experimental protocols to validate these claims and discuss the structural conditions under which a failure-inclusive publishing culture could emerge.

In brief

The case here is argued, not measured: publication bias toward positive results is claimed to propagate into LLMs, which inherit a distorted prior on how often experiments succeed, just as the corpus of unpublished failure becomes the largest untapped source of human-generated training text.

The paper is a conceptual framework assembling existing literature on the "file drawer" with observations about LLM training and peer review. Supporting figures are borrowed: Fanelli's analysis of over 4,600 papers found positive-result rates exceeding 80% after 1999, peaking at 88.6% in 2005; Registered Reports in psychology drop positive rates from 96% (N=152) to 44% (N=71); 93% of 82 ML-for-PDE papers claimed superiority over numerical baselines, with 79% of those claims resting on weak baselines; Chalmers and Glasziou estimated 85% of research investment is avoidably wasted. On the review side, ICLR submissions went from roughly 7,000 in 2024 to nearly 20,000 in 2026, and 21% of 75,800 ICLR 2026 reviews were flagged as entirely AI-generated.

No experiment was run. Two are proposed: predicting outcomes for 200–500 ClinicalTrials.gov trials from pre-trial information, and a 5-condition context ablation (no context, success-only, failure-only, both, classified) with a dose-response curve at 10, 50, 100, 200 failure records.

The authors concede the core claim rests partly on analogy to human expertise, that the proposed Type A/B/C failure taxonomy is untested, and that incentives are unaddressed. Cited empirical support for learning from failure is narrow: chemical reaction prediction and sparse-reward generative tasks.

Treat this as an agenda and a set of hypotheses to test, not evidence that failure data calibrates models.

Written from the abstract by claude-opus-5 on 2026-10-24, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsLanguage models trained on the published literature inherit its positive bias, because failure data is systematically missing from what they read, and that degrades them in three roles at once — as research tools that suggest what to try, as consumers of scarce high-quality training data, and as peer reviewers — so the absence of negative results now costs more than it did when only people read the papers.