agent-time-horizon-at-50-percent-reliability-doubles-about-every-seven-months CLM-0019observationsingle paperReviewed by AI
On 170 software and research tasks timed with human domain experts, the length of task that frontier agents complete at 50% reliability, measured in human time, has doubled roughly every seven months since 2019 and was around 110 minutes for o3 in mid-2025 — but the horizon at 80% reliability follows the same trend at roughly five times shorter, and success drops as tasks carry more of the 16 "messiness" factors that separate benchmark tasks from real work, so the headline horizon overstates what an agent can be relied on to finish.
Capability: Completing tasks that take many steps or many hours
Observed on
2019–2025 frontier models (GPT-2 through o3 and Claude 3.7), on HCAST, RE-Bench and SWAA tasks with human baselines. Coding agent.
Sources
- Figure 1 and Section 3: 50% time horizon doubling approximately every 7 months over 6 years, possibly faster since 2024; Figure 17 for the 80% horizon being roughly 5x shorter; Section 2.1 for the 170 tasks and the messiness factors. Fitted trend with bootstrapped confidence intervals, not a controlled experiment.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Across 393 model–benchmark observations in ten capability domains for models released between 2023 and September 2026, the share of benchmark headroom closed is uneven: bounded, readily verified domains are near their ceilings (advanced mathematics 86.4 and graduate-level science 85.8 on a 0–100 index where 0 is the benchmark's entry-year frontier), while interactive, stateful ones lag (software engineering 52.6, search and terminal agents 56.8, tool agents 39.9) — so gains in bounded environments have not transferred uniformly to long, stateful workflows where errors propagate across the trajectory.Completing tasks that take many steps or many hours · reviewed by AI
- For single-agent procedural tasks where the relevant state fits a fixed schema known in advance, replacing the appended conversation transcript with a structured execution state that the model rewrites each step — discarding the reasoning trace once the state patch is validated — holds accuracy flat as the horizon grows while history-appending runtimes degrade, and the gain comes from the structure rather than the shorter prompt.Completing tasks that take many steps or many hours · reviewed by AI
- For long-horizon tool-calling agents (tasks needing five or more calls), embedding state-transition cues — preconditions, invariants, completion states — into the descriptions of the tools on the intended chain is what actually steers the agent's trajectory; runtime corrective text appended to tool results only patches residual drift, and the plausible user prompt alone (persona, deadlines, format constraints) does not establish the trajectory at all.Following instructions hidden in data · reviewed by AI
- In long-horizon tool-mediated environments where strategically relevant state is available only on explicit query (300+ turn Civilization VI via MCP, 23 runs, four model families under a shared playbook), agents allocate only about 1-2% of non-infrastructure tool calls to strategic monitoring and execute only about 48-66% of the near-term commitments they state in their own planning diaries, even though the playbook told them to query victory progress every 20 turns.Completing tasks that take many steps or many hours · reviewed by AI
- In file-based sandbox agent tasks with distractor artefacts, scoring only whether the task goal is met overstates reliability, because agents frequently complete the target while modifying unrelated files: across nine LLM agents, pass rate exceeds a success-plus-preservation metric by roughly 10-17 points, with the gap widening as model capability falls.Completing tasks that take many steps or many hours · reviewed by AI
Notes
Filed the day Russ forwarded it, as the anchor for the long-horizon execution capability. The interesting number is not the doubling time but the ratio between the 50% and 80% horizons: reliability, not capability, is what limits handing an agent a long task. The extrapolation to month-long tasks is the authors' and rests on the trend continuing; treat it as a forecast, not a finding.