Whether the measurement made the finding

Claims

• mechanismsingle paper↑1↓0
  When each capability domain's annual benchmark frontier is normalised to its own scale (0 at the benchmark's entry-year frontier, 100 at a perfect score), the domains move on different curves — broad knowledge gains slow from 32.8 to 26.9 to 17.6 points a year, legal reasoning from 48.2 to 11.4 to 4.9, while advanced mathematics accelerates from 32.8 to 53.6 and multimodal reasoning jumps 59.7 then only 2.5 — so a single aggregate benchmark score conceals both the level and the trajectory shape of progress, and cannot say which capabilities are saturating.
• mechanismsingle paper↑1↓0
  For behaviour-preserving whole-repository tasks such as stack migrations, where the starting repository already passes its test suite, a fixed behavioural test suite cannot distinguish a completed migration from an untouched or shim-wrapped repository, so adding more tests does not close the gap and a non-behavioural audit with veto power is required.
• mechanismsingle paper↑1↓0
  For code generation benchmarks, deciding whether a benchmark item leaked into a model's training set is done better by a classifier over code-specific signals (test pass rate, CodeBLEU, edit distance, perplexity, CodeBERT embeddings of generated vs reference code) than by perplexity alone, because memorized items show abnormally high functional correctness and similarity to the reference solution, not just low perplexity.
• mechanismsingle paper↑1↓0
  When comparing a weight-shared recursive model against a depth-matched plain transformer on code generation, the ranking read from a single checkpoint can invert during training, because the larger-parameter arm passes its validation optimum and overfits while the parameter-lean recursive arm keeps improving — so a late-epoch "recursion fits better" reading is a degradation artifact, not a capability difference.
• mechanismsingle paper↑1↓0
  Evaluations that define "the model did not think" by a disabled thinking mode or by the absence of a long trace measure the wrong thing: a disabled mode may still emit reasoning in the answer, and a long trace may hold filler rather than inference; separating the pre-answer text from the answer and scoring it for emptiness, relevance to the question and explicit inference distinguishes answer-only output, relevant but non-inferential text, and genuine reasoning, which the proxies conflate.
• observationsingle paper↑1↓0
  On function-level algorithmic tasks translated to C#, a model's functional test pass rate carries almost no information about the structural quality or runtime cost of the code it writes: across 340 solutions from four frontier models, correctness correlated near zero with a composite quality score (r = 0.075) and model rankings by correctness reversed against rankings by quality, while the structurally simplest generator was also the most memory-hungry.
• mechanismsingle paper↑1↓0
  When grading free-form answers to algorithm-analysis and complexity-theory questions against instructor reference answers, n-gram overlap metrics carry almost no signal about answer quality, because logically equivalent proofs differ in notation, variable names and proof strategy: BLEU-4 was 0.0000 on all 179 questions and ROUGE-1 F1 stayed in a narrow 0.088-0.129 band across topics while a rubric score varied from 0.66 to 0.83.
• observationsingle paper↑1↓0
  When grading LLM-generated unit tests by pass rate and line coverage alone, a share of passing test suites never directly call the function under test — they reach it through wrappers, mocks, or unrelated public APIs — so pass rate overstates test quality, and this gap widens under agentic repository-access workflows and richer file-level context.
• mechanismsingle paper↑1↓0
  When a hallucination metric is an unsupported-claim rate computed over whatever claims a model happens to make, models that avoid committing to verifiable claims or directional outputs score best, so the ranking partly measures abstention rather than grounding — on financial chart-to-recommendation tasks the lowest unsupported-claim model made directional recommendations on only 6.4% of queries and ranked near the bottom once coverage was factored in.
• mechanismsingle paper↑6↓0
  A large share of an agent benchmark score is attributable to the harness around the model rather than the model itself: holding the model fixed and changing only guides, sensors, retries and state management is reported to move the same model from 30.9% to 74.6% on GAIA and 30th to 5th on Terminal Bench — so a score reported for "a model" on an agentic benchmark is a score for a model-plus-harness, and comparisons across labs with different harnesses are not comparisons of models.
• mechanismmechanism reasoning↑2↓0
  Reporting how much a lab's models are used in its own research — tokens, lines of code, inference compute, experiments per researcher — cannot show that the models are accelerating the research, because every one of those is an input. Establishing the loop needs an outcome variable over time: algorithmic efficiency as a function of capability, which is the edge that turns a pipeline into a feedback loop.
• mechanismmechanism reasoning↑1↓0
  Language models trained on the published literature inherit its positive bias, because failure data is systematically missing from what they read, and that degrades them in three roles at once — as research tools that suggest what to try, as consumers of scarce high-quality training data, and as peer reviewers — so the absence of negative results now costs more than it did when only people read the papers.
• mechanismsingle paper↑1↓0
  When an LLM judge is validated against expert annotators on multi-dimensional rubrics for open-ended agent outputs, a single pooled agreement figure overstates reliability: per-dimension human-human agreement varies widely (substantial on a recall-style Completeness dimension, only slight to fair on Relevancy, Structural Clarity and Scope Adherence), the pooled statistic is inflated by between-dimension variation, and the judge's residual disagreements are directional (consistently one step stricter on Completeness, one step more generous on Structural Clarity) rather than noise.
• mechanismmechanism reasoning↑1↓0
  Machine learning research treats beating the state of the art on a benchmark as the main kind of publishable result, so negative results — methods that did not work, conditions under which a known method fails — go largely unpublished, and compute is spent repeating attempts that others have already found not to work; registering what does not work is no less important than registering what works.
• observationsingle paper↑1↓0
  On the best current calibration, AI is not yet accelerating its own development in a self-sustaining way: the modelled threshold is that a one-unit gain in model capability must buy at least 15% higher AI R&D productivity, and the back-of-envelope figure from reported engineer uplift since coding agents launched is about 9% — below it, but rising, so the gap is a current reading rather than a ceiling.
• mechanismmechanism reasoning↑1↓0
  In multi-hop KGQA evaluated as graph navigation with annotated gold evidence paths, reported agreement between an agent's traversed path and the reference path cannot be credited to question conditioning unless it is compared against question-agnostic references — unbiased random walks and answer-oracle shortest paths — because dense or small graphs let structure alone deliver both answer reachability and partial path overlap.
• mechanismsingle paper↑1↓0
  In multi-turn LLM buyer-seller market simulations, a measured welfare gain attributed to a platform guardrail can be mostly an artifact of the scaffold the guardrail was bundled with: holding the offer schema and buyer choice rule identical across guarded and unguarded cells shrank or reversed the effect (+87.4, +35.0, +28.8 became +7.2, -13.9, +23.8 across a Qwen2.5 1.5B-14B ladder), and single-generation effects on the most favorable profiles shrank from a mean +229 to +37.6 with three generations per cell, with generation residuals accounting for 49.9% of variation.
• mechanismsingle paper↑1↓0
  On multi-turn tool-calling benchmarks that score whether the final environment state matches a gold trajectory, aggregate accuracy hides wrong action-class choices: a model can call a withheld or under-specified tool with fabricated arguments, be handed the missing piece on a later turn, and still be graded PASS — so heavily tool-trained model families rank high while almost never asking for a missing parameter or refusing an unavailable function.
• observationsingle paper↑1↓0
  When NL-to-PDDL problem generation is scored by operational acceptance (parses, conforms to domain, a planner finds a plan, VAL validates it, an LLM critic accepts), the score overstates faithfulness to the described task, and the gap widens for PDDL 2.1 temporal and numeric problems where operationally accepted instances can almost never reconstruct the curated reference encoding.
• mechanismsingle paper↑1↓0
  When research-idea forecasts are scored by whether a later paper matches them under an LLM judge rubric, higher scores partly reflect broader, less specific ideas: the backbone with the higher hit rate also produced measurably more general forecasts and had more retrieved candidate papers pass the gate, and tightening the specificity threshold does not separate the two.
• observationsingle paper↑1↓0
  On span-level hallucination-detection leaderboards where absolute scores are low, bootstrap resampling of the test set moves top systems across wide rank intervals, so point-estimate ordering does not establish that one detector beats another.
• observationsingle paper↑1↓0
  On a suite of verifier-checked tool-using ReAct task families (arithmetic, SQL analytics, security judgment, URL grounding, dependency planning, orchestration, tool policy), a single task-weighted aggregate accuracy hides inverted per-family rankings: the model that ranks last overall can lead both URL-grounding families while scoring lowest on structured SQL and multi-step tool families, which spread the four tested models by more than sixty points.
• mechanismsingle paper↑1↓0
  In synthetic hallucination benchmarks where faithful items are human-written and hallucinated items are LLM-rewrites, a detector's binary score partly measures human-vs-machine style rather than factuality: feeding faithful LLM rewrites of the same truthful articles makes some fine-tuned encoders flag nearly all of them as hallucinated.
• observationsingle paper↑1↓0
  For text-to-CAD program generation, final-geometry similarity (IoU against a reference solid) carries almost no information about whether the generated program preserves the intended construction process: on multi-entity Text2CAD samples with Qwen3, sample-level IoU and a constraint-satisfaction rate over expected entities, Boolean roles, sketch properties and inter-entity relations correlate near zero (Pearson r = 0.022, Spearman 0.023), and high-IoU programs can omit expected entities or invert additive/subtractive roles.
• observationreplicated↑2↓0
  The published research literature overreports results that support the hypothesis tested: in psychology, 96% of a random sample of standard hypothesis-testing papers reported a positive first result against 44% of Registered Reports, which are accepted before results are known; and across all disciplines the share of hypothesis-testing papers reporting positive support rose from 70.2% in 1990-1991 to 85.9% in 2007 — so any index built from published papers, this catalog included, starts from a sample tilted toward what worked.
• observationsingle paper↑1↓0
  For training-free inference-time hallucination mitigation in 7B vision- language models, reported gains on object-hallucination benchmarks are largely inseparable from reduced informativeness: hallucination rate and object recall/coverage move together, so a lower score can mean the model mentioned fewer visual entities rather than grounded better.

Techniques

• Detect benchmark leakage with a code-specific membership claprocess
  Fine-tune a shadow model on part of a benchmark to label members and non-members, then train a classifier on expert features (pass rate, CodeBLEU, edit distance, perplexity) fused with CodeBERT embeddings to predict whether a benchmark item
• Deterministic checks before LLM judgesprocess
  Wire linters, test suites and schema validators into the agent's loop before adding any model-based judgment.
• Report operational acceptance and reference reconstruction sprocess
  Use parser, domain-conformance, planner, VAL and LLM-critic checks as the online stopping criterion, but compute renaming-invariant structural match and semantic equivalence against curated reference problems offline as a separate benchmark
• Report per-family spread, not an aggregate agent scoreprocess
  Split an agent benchmark into skill-isolated families with verifier-computed gold (canonical SQL, exact DAG schedule, closed-form lambda) and report per-family accuracy plus token cost from one instrumented run instead of a blended score.
• Score proof answers with a binary pedagogical rubric, not n-process
  Replace lexical-overlap scoring with the mean of six binary per-response criteria (step enumeration, notation correctness, worked example, sub-question coverage, explanation depth, proof-step granularity).
• Unify offer schema and chooser, then flag the treatmentprocess
  Apply the policy treatment as post-parsing boolean flags over one shared message schema and one shared buyer decision rule, average repeated generations within profile-condition, and bootstrap over profiles rather than generation rows.

Sources

Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.

• Thinking Inertia: LLMs Keep Thinking When Told Not To2026-10-08
  ↑ supports a claim here
• AlgoRAG: Retrieval-Augmented Generation for Theoretical Computer Science Education -- A Comprehensive Evaluation Framework for Algorithm Analysis and Complexity Theory2026-09-13
  ↑ supports a claim here
• E2A-Bench: Benchmarking Evidence-to-Action Reliability in Financial Chart Reasoning2026-09-13
  ↑ supports a claim here
• Theseus in the Graph: Towards Traceable Multi-Hop Graph Navigation2026-09-13
  ↑ supports a claim here
• The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement2026-09-10
  ↑ supports a claim here
• Keep Evaluation Fair: Detecting Data Leakage in Code Generation Benchmarks via Membership Inference Attacks2026-09-09
  ↑ supports a claim here
• Grounded Evaluation and Repair for NL-to-PDDL Problem Generation2026-09-09
  ↑ supports a claim here
• CIT-CAD: Constraint Intent Tree-based CAD Code Generation and Verification2026-09-07
  ↑ supports a claim here
• On OpenAI's research-acceleration disclosure — the measured variables are all inputs2026-09-06 · post
  ↑ supports a claim here
• When Guardrails Look Effective: Construct Validity Failures in LLM Agent Commerce Evaluation2026-09-01
  ↑ supports a claim here
• Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Calling2026-09-01
  ↑ supports a claim here
• Can Large Language Models Forecast What Researchers Study Next?2026-09-01
  ↑ supports a claim here
• Does Playing it Safe Count as Faithfulness? Reassessing LVLM Hallucination Mitigation Methods2026-09-01
  ↑ supports a claim here
• Evaluating Tiny Recursive Models Across Training for Code Generation2026-08-29
  ↑ supports a claim here
• BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click2026-08-27
  ↑ supports a claim here
• XREPOTEST: Benchmarking Multilingual Repository-Level Unit Test Generation for Large Language Models2026-08-26
  ↑ supports a claim here
• Overview of SHROOM-Visions 2026: A Shared Task on Hallucination Detection in Large Vision-Language Models2026-08-26
  ↑ supports a claim here
• Lost in Speech: Trilingual Spoken Hallucination Detection Across Audio and Transcripts2026-08-25
  ↑ supports a claim here
• SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?2026-08-24
  ↑ supports a claim here
• Benchmarking the Titans: A Multi-Dimensional Empirical Evaluation of LLM Code Generation Quality in the .NET Ecosystem2026-08-23
  ↑ supports a claim here
• Designing a Robust LLM-Based Evaluation System for Agentic AI in Drug Discovery Through Human Alignment2026-08-21
  ↑ supports a claim here
• Harness Engineering — Agent = Model + Harness: The 6-Layer Production Playbook2026-08-01 · post
  ↑ supports a claim here
• The Economics of Recursive Self-Improvement2026-07-13
  ↑ supports a claim here · ↑ supports a claim here
• Agentic Harness Engineering: The Operating System for Machine Intelligence2026-06-27 · post
  ↑ supports a claim here
• Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows2026-05-27
  ↑ supports a claim here
• LLMs Have Made Failure Worth Publishing2026-04-04
  ↑ supports a claim here
• I Improved 15 LLMs at Coding in One Afternoon. Only the Harness Changed.2026-02-12 · post
  ↑ supports a claim here
• Position: Embracing Negative Results in Machine Learning2024-06-06
  ↑ supports a claim here
• An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports2021-04-01
  ↑ supports a claim here
• Negative results are disappearing from most disciplines and countries2011-09-11
  ↑ supports a claim here
• HAL GAIA Leaderboard (Holistic Agent Leaderboard, Princeton)2026 · post
  ↑ supports a claim here
• Improving Deep Agents with harness engineering2026 · vendor-doc
  ↑ supports a claim here