benchmark-headroom-closes-fastest-in-bounded-domains-and-lags-in-stateful-agent-work CLM-0025observationsingle paperReviewed by AI
Across 393 model–benchmark observations in ten capability domains for models released between 2023 and September 2026, the share of benchmark headroom closed is uneven: bounded, readily verified domains are near their ceilings (advanced mathematics 86.4 and graduate-level science 85.8 on a 0–100 index where 0 is the benchmark's entry-year frontier), while interactive, stateful ones lag (software engineering 52.6, search and terminal agents 56.8, tool agents 39.9) — so gains in bounded environments have not transferred uniformly to long, stateful workflows where errors propagate across the trajectory.
Capability: Completing tasks that take many steps or many hours
Observed on
Frontier models 2023–2026 scored on public benchmarks per domain, normalised as the Headroom-Closed Index; cybersecurity segments caveated for changed task subsets. Autonomous agent.
Sources
- Section 2.1, Figure 3, Observation 2: HCI values by domain for 2026 and the 33.2, 29.1 and 45.9-point gaps to graduate-level science. The authors caveat the cybersecurity trajectory (91.9) for changed Cybench subsets and pass@1 aggregation. Descriptive aggregation of published benchmark results, not a controlled comparison.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On 170 software and research tasks timed with human domain experts, the length of task that frontier agents complete at 50% reliability, measured in human time, has doubled roughly every seven months since 2019 and was around 110 minutes for o3 in mid-2025 — but the horizon at 80% reliability follows the same trend at roughly five times shorter, and success drops as tasks carry more of the 16 "messiness" factors that separate benchmark tasks from real work, so the headline horizon overstates what an agent can be relied on to finish.Completing tasks that take many steps or many hours · reviewed by AI
- For long-horizon tool-calling agents (tasks needing five or more calls), embedding state-transition cues — preconditions, invariants, completion states — into the descriptions of the tools on the intended chain is what actually steers the agent's trajectory; runtime corrective text appended to tool results only patches residual drift, and the plausible user prompt alone (persona, deadlines, format constraints) does not establish the trajectory at all.Following instructions hidden in data · reviewed by AI
- For single-agent procedural tasks where the relevant state fits a fixed schema known in advance, replacing the appended conversation transcript with a structured execution state that the model rewrites each step — discarding the reasoning trace once the state patch is validated — holds accuracy flat as the horizon grows while history-appending runtimes degrade, and the gain comes from the structure rather than the shorter prompt.Completing tasks that take many steps or many hours · reviewed by AI
- On 450 Lean 4 theorem targets in graduate stochastic processes — a field thinly covered by Mathlib — a tool-using frontier agent (Opus 4.8 with Lean LSP access, one 15-minute run per problem) proves only 34.9%, and the rate varies widely by topic, from about 5% on renewal processes to about 62% on martingales and stopping.Formalizing and machine-checking mathematics · reviewed by AI
- In long-horizon tool-mediated environments where strategically relevant state is available only on explicit query (300+ turn Civilization VI via MCP, 23 runs, four model families under a shared playbook), agents allocate only about 1-2% of non-infrastructure tool calls to strategic monitoring and execute only about 48-66% of the near-term commitments they state in their own planning diaries, even though the playbook told them to query victory progress every 20 turns.Completing tasks that take many steps or many hours · reviewed by AI
Notes
Sits beside the METR time-horizon claim: that one measures how long a task agents complete, this one measures how far each domain's benchmarks have been closed, and both say the same thing about long, stateful work. The index depends on which benchmarks exist per domain and on their entry-year frontiers, so the numbers are comparable within this paper's construction, not as absolutes.