On realistic web tasks with explicit goals, the best model completed only a small fraction end to end, far below human performance.

Capability: Following an unfamiliar procedure

Sources

supportsWebArena: A Realistic Web Environment for Building Autonomous Agents
On realistic web tasks with explicit goals, the best model completed only a small fraction end to end, far below human performance.

Reviewed by AI and Russ MillerLast checked: 2026-09-03Evidence activity

Either way, bring evidence: a link, or what you saw.

Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.

Related claims

For LLM web agents on cluttered pages containing many concurrent interface components, success falls sharply as component count grows, and having the agent first emit a localized sub-goal plus a task-relevant sub-observation before choosing an action recovers much of the loss, because self-assessment of the action is more reliable in the reduced context.Using the tools it is given · reviewed by AI
On 170 software and research tasks timed with human domain experts, the length of task that frontier agents complete at 50% reliability, measured in human time, has doubled roughly every seven months since 2019 and was around 110 minutes for o3 in mid-2025 — but the horizon at 80% reliability follows the same trend at roughly five times shorter, and success drops as tasks carry more of the 16 "messiness" factors that separate benchmark tasks from real work, so the headline horizon overstates what an agent can be relied on to finish.Completing tasks that take many steps or many hours · reviewed by AI
On web-browsing (WebArena, Mind2Web) and software-engineering (SWE-Bench-Verified) benchmarks, an agent that distills reusable strategies from both its successful and its failed attempts, judged by an LLM without ground truth, and retrieves the relevant ones on later tasks, outperforms memories that store raw trajectories or only successful routines — up to 20% relative improvement in success and up to 16% fewer interaction steps — with the weights unchanged; including the failures is part of the gain, not noise.Improving its own scaffolding from failures · reviewed by AI
On 430 tasks distilled from real professional-forum threads across seven domains, the best model (GPT-5) scores about 54%, under the 60% passing line the authors set, with computer science and applied technology lowest at roughly 40-45%, so long-tail practitioner knowledge remains a gap even where standard benchmarks are saturated.Competence in a specialist domain · reviewed by AI
Across 393 model–benchmark observations in ten capability domains for models released between 2023 and September 2026, the share of benchmark headroom closed is uneven: bounded, readily verified domains are near their ceilings (advanced mathematics 86.4 and graduate-level science 85.8 on a 0–100 index where 0 is the benchmark's entry-year frontier), while interactive, stateful ones lag (software engineering 52.6, search and terminal agents 56.8, tool agents 39.9) — so gains in bounded environments have not transferred uniformly to long, stateful workflows where errors propagate across the trajectory.Completing tasks that take many steps or many hours · reviewed by AI