arxiv-2406-03980 SRC-0581 · paper

Position: Embracing Negative Results in Machine Learning

Florian Karl, Lukas Malte Kemeter, Gabriel Dax, Paulina Sierak

Created: 2024-06-06 · Ingested: 2026-10-19

Publications proposing novel machine learning methods are often primarily rated by exhibited predictive performance on selected problems. In this position paper we argue that predictive performance alone is not a good indicator for the worth of a publication. Using it as such even fosters problems like inefficiencies of the machine learning research community as a whole and setting wrong incentives for researchers. We therefore put out a call for the publication of "negative" results, which can help alleviate some of these problems and improve the scientific output of the machine learning research community. To substantiate our position, we present the advantages of publishing negative results and provide concrete measures for the community to move towards a paradigm where their publication is normalized.

In brief

Predictive performance on chosen benchmarks is a weak and noisy proxy for the worth of a machine learning paper, and the near-exclusive publication of positive results makes the field less efficient. The argument is a position paper, not a study: the authors define a negative result as failure to reject the null that a proposed method beats existing methods, and split it into novel-method negative results (NMNR) and existing-method negative results (EMNR), the latter already better tolerated.

Evidence is entirely secondhand citation. Roberts et al. (2021) screened 2,212 studies on COVID detection and prognosis, scrutinised the most promising 62, and found none clinically usable. Varoquaux and Cheplygina report Kaggle winners whose margins were smaller than evaluation noise. Growth figures cited: AI publications more than doubled between 2010 and 2021 to almost 500,000, with a doubling rate of roughly 23 months and over 4,000 papers per month in 2023.

Nothing is measured here. There is no survey of reviewers, no audit of acceptance rates by result sign, no estimate of how much duplicated effort negative-result publication would save. The 4 counterarguments the authors raise are answered by argument, largely by appeal to a healthy review process.

The contribution is 8 concrete proposals, including two-phase review where results are withheld in round 1, incentivised replication studies, and venue-level criteria for judging negative results.

Read it as agenda-setting for how your venue reviews, not as a finding about models.

Written from the abstract by claude-opus-5 on 2026-10-24, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsMachine learning research treats beating the state of the art on a benchmark as the main kind of publishable result, so negative results — methods that did not work, conditions under which a known method fails — go largely unpublished, and compute is spent repeating attempts that others have already found not to work; registering what does not work is no less important than registering what works.