arxiv-2605-00334 SRC-0417 · paperAgentFloor: How Far Up the tool use Ladder Can Small Open-Weight Models Go?
Ranit Karmakar, Jayita Chatterjee
Created: 2026-05-01 · Ingested: 2026-09-26
https://arxiv.org/abs/2605.00334(opens in a new tab)Production agentic systems make many model calls per user request, and most of those calls are short, structured, and routine. This raises a practical routing question that existing evaluations do not directly answer: which parts of an agent workflow truly require large frontier intelligence, and which can be handled by smaller models? We introduce AgentFloor, a deterministic 30-task benchmark organized as a six-tier capability ladder, spanning instruction following, tool use, multi-step coordination, and long-horizon planning under persistent constraints. We evaluate 16 open-weight models, from 0.27B to 32B parameters, alongside GPT-5 across 16,542 scored runs. Our results reveal a clear boundary of model necessity. Small and mid-sized open-weight models are already sufficient for much of the short-horizon, structured tool use work that dominates real agent pipelines, and in aggregate, …
In brief
Small open-weight models handle the short, structured tool calls that make up most agent traffic, and the frontier advantage is confined to long-horizon planning under persistent constraints — where neither side is reliable.
AgentFloor is a deterministic 30-task benchmark, 5 tasks per tier across 6 tiers (no-tool instruction following, single tool, 2-tool chain, branching, multi-source synthesis, long-horizon planning), over 8 abstract tools and an in-memory fixture database with no live APIs or filesystem. 16 open-weight models from 0.27B to 32B served via Ollama plus GPT-5 were run at temperature 0 with native tool calling only, 16,542 scored runs total.
Paired across 30 tasks (n = 270), gemma4:26b and GPT-5 are equivalent at a pre-registered ±10 pp margin (Δ = +0.4 pp, 90% CI [−4.0, +5.1]), at about 15× cheaper per passed task on Mac self-hosting, 3× on cloud GPU, and 2.5× faster. Open-weight is strictly higher on A0 (100% vs 80%, +20.0 pp) and GPT-5 strictly higher on E (10% vs 0%, Δ = −10.2 pp).
Evidence uses pre-registered TOST, bootstrap CIs (10,000 resamples), Holm correction, and a priority failure cascade. The authors state plainly that per-tier samples (n = 45) cannot certify equivalence on B, C, or D despite point estimates within 9 pp.
Ablations show no universal lever: explicit-submission gave +100 pp on 1 model on 1 task and was null elsewhere; a plan/execute/submit prompt regressed every model tested (ministral-3:8b −33 pp). Tier-E failure mass differs by model (GPT-5 F5 39%, F5b 24%; gemma4:26b F4 41%, F1 36%).
Boundaries: 1 frontier anchor, 5 tasks per tier, abstract tools only, no human baseline. Use it as a routing prior, not a live-agent forecast.
Written from the abstract by claude-opus-5 on 2026-10-24, with every figure checked against it. Not a substitute for the paper.
Referenced by
Claims in this catalog that draw on this source, and whether as support or counterpoint.
- contestsAcross eight interactive environments, models fail mostly on long-horizon tool interaction, with open models far behind commercial ones.
- supportsOn a 30-task ladder spanning instruction following, tool use, multi-step coordination and long-horizon planning, evaluated over 16 open-weight models from 0.27B to 32B against GPT-5 across 16,542 scored runs, small and mid-sized open models are already sufficient for the short-horizon structured tool use that makes up most calls in real agent pipelines, and the strongest open-weight model matches GPT-5 in aggregate — but the gap reappears on long-horizon planning that needs sustained coordination and constraint tracking, where frontier models still lead and neither side is reliable, and the boundary is not explained by model scale alone.