small-open-models-suffice-for-routine-agent-calls-not-for-long-horizon-planning CLM-0337observationsingle paperReviewed by AI
On a 30-task ladder spanning instruction following, tool use, multi-step coordination and long-horizon planning, evaluated over 16 open-weight models from 0.27B to 32B against GPT-5 across 16,542 scored runs, small and mid-sized open models are already sufficient for the short-horizon structured tool use that makes up most calls in real agent pipelines, and the strongest open-weight model matches GPT-5 in aggregate — but the gap reappears on long-horizon planning that needs sustained coordination and constraint tracking, where frontier models still lead and neither side is reliable, and the boundary is not explained by model scale alone.
Capability: Using the tools it is given
Observed on
2026, 16 open-weight models 0.27B-32B and GPT-5 on the authors' own AgentFloor benchmark. Autonomous agent.
Applies when
- Effect on the capability:
- helps, narrowly
- Needs:
- a second model with its own context
- Helps most:
- independent
- Cost:
- low
- Fails when:
- The step needs long-horizon planning or constraint tracking over many steps, where the small model's shortfall reappears; and the routing rule itself does not transfer, since the paper finds failures respond to model-specific rather than universal interventions.
Sources
- The paper's own benchmark, harness and sweep, released with the run corpus. The aggregate is an average over a ladder whose lower tiers are short structured calls by construction, so "matches GPT-5" is a statement about that mix, not about frontier-level capability.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For open-source LLM agents driving a ROS robot, having the model emit a pseudo-code tool-call plan up front, storing it in a persistent scratchpad, and then restricting each tool invocation to the plan's next step raises task success most on the longest multi-step tasks and for the smaller open models (GPT-OSS-20B on the hardest task rose from 35% with a persistent plan alone to 55% with the constraint), while a strong proprietary model gains little.Completing tasks that take many steps or many hours · reviewed by AI
- On four diagram-grounded geometry benchmarks a multi-agent pipeline consistently helps open-source vision-language models (Qwen-2.5-VL 7B +6.8 and 32B +3.3 points on Geometry3K, with further gains on OlympiadBench and We-Math) but a strong closed model (Gemini 2.0 Flash) generally did better as a single agent on the classic benchmarks, gaining only modestly on the newest one, so agentic decomposition pays for weaker models and less familiar tasks rather than universally.Getting more from several models than from one · reviewed by AI
- For long-horizon tool-calling agents (tasks needing five or more calls), embedding state-transition cues — preconditions, invariants, completion states — into the descriptions of the tools on the intended chain is what actually steers the agent's trajectory; runtime corrective text appended to tool results only patches residual drift, and the plausible user prompt alone (persona, deadlines, format constraints) does not establish the trajectory at all.Following instructions hidden in data · reviewed by AI
- Across eight interactive environments, models fail mostly on long-horizon tool interaction, with open models far behind commercial ones.Using the tools it is given · contested
- In multi-agent LLM workflows with a mixed-size model zoo, choosing which model instantiates each agent step-by-step from an online estimate of task-progress gain plus remaining time and cost budget keeps task quality near the best single-model policy at lower energy than one-shot query-level routing or stall-triggered escalation.Getting more from several models than from one · reviewed by AI
Notes
Filed with the technique it argues for, model-routing, and as the contesting source on CLM-0012, which held that open models are far behind commercial ones on agent work. Single benchmark, designed by the authors who report the result, which is the main reason to hold it lightly.