across-eight-interactive-environments-models-fail-mostly-long-horizon CLM-0012mechanismsingle papercontestedReviewed by AI and Russ Miller
Across eight interactive environments, models fail mostly on long-horizon tool interaction, with open models far behind commercial ones.
Capability: Using the tools it is given
Sources
- Across eight interactive environments, models fail mostly on long-horizon tool interaction, with open models far behind commercial ones.
- Contests the second half only. On a 30-task ladder over 16,542 scored runs, the strongest open-weight model matches GPT-5 in aggregate, and small-to-mid open models suffice for the short-horizon structured tool use that dominates real pipelines. It agrees that long-horizon planning is where the gap remains, and adds that neither side is reliable there.
Suspected axis of disagreement(a guess, not verified)
Task horizon, and how the aggregate is taken. The supporting benchmark scores whole interactive episodes, where open models trailed in 2023; the contesting one scores a ladder whose lower tiers are short structured calls by construction, on 2026 open weights, and reports an aggregate across all tiers. Both agree open models lag on long-horizon planning, so the disagreement may be about what "agent work" averages over, and about two and a half years of open-weight progress, rather than about a fact.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On a 30-task ladder spanning instruction following, tool use, multi-step coordination and long-horizon planning, evaluated over 16 open-weight models from 0.27B to 32B against GPT-5 across 16,542 scored runs, small and mid-sized open models are already sufficient for the short-horizon structured tool use that makes up most calls in real agent pipelines, and the strongest open-weight model matches GPT-5 in aggregate — but the gap reappears on long-horizon planning that needs sustained coordination and constraint tracking, where frontier models still lead and neither side is reliable, and the boundary is not explained by model scale alone.Using the tools it is given · reviewed by AI
- For locally deployed open-weight models driving a stateful, dependency-ordered MCP tool server, cutting tool descriptions from full specifications (purpose, parameter semantics, constraints, failure conditions) down to one sentence each raises the fraction of calls the server rejects for every model tested, while its effect on task coverage is less consistent.Using the tools it is given · reviewed by AI
- For long-horizon tool-calling agents (tasks needing five or more calls), embedding state-transition cues — preconditions, invariants, completion states — into the descriptions of the tools on the intended chain is what actually steers the agent's trajectory; runtime corrective text appended to tool results only patches residual drift, and the plausible user prompt alone (persona, deadlines, format constraints) does not establish the trajectory at all.Following instructions hidden in data · reviewed by AI
- Fine-tuning on a large corpus of real API-call traces markedly improves multi-tool task completion in open models.Using the tools it is given
- Across many agent environments, most tested models scored below sixty percent on safety, with failures concentrated in unsafe tool actions.Prioritizing safety under conflicting goals