a-constrained-agent-computer-interface-beats-raw-shell-access CLM-0006mechanismsingle paperReviewed by AI
Giving a coding agent a raw shell is worse than giving it a designed interface: SWE-agent's constrained Agent-Computer Interface — a file viewer windowed to 100 lines, a search tool that truncates large output, and an editor that lints before a change persists — roughly doubled SWE-bench resolution over raw shell access with the same model, because verbose output and unmanaged state are where the model loses its place.
Capability: Using the tools it is given · Agentic, Coding agent
Observed on
2024, GPT-4-class models on SWE-bench.
Model cards: GPT-4
Sources
- The paper's central result and the origin of the term agent-computer interface. The comparison holds the model fixed and varies the interface.
- The same effect two years later from one narrow change: tagging each line with a content hash so edits cannot silently hit the wrong line improved nearly all of 16 models, Grok Code Fast 1 from 6.7% to 68.3%.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In file-based sandbox agent tasks with distractor artefacts, scoring only whether the task goal is met overstates reliability, because agents frequently complete the target while modifying unrelated files: across nine LLM agents, pass rate exceeds a success-plus-preservation metric by roughly 10-17 points, with the gap widening as model capability falls.Completing tasks that take many steps or many hours · reviewed by AI
- Giving a coding agent a repository context file — AGENTS.md, CLAUDE.md — does not raise its success rate on benchmark coding tasks and costs about 20% more inference: across 4 agents, 2 benchmarks and 3 conditions, LLM-generated files hurt slightly in 5 of 8 settings while developer-written ones gained 2.4% (p=0.21), and agents obey the files — which is why they spend more — so the files do not carry success-relevant information rather than being ignored.Following an unfamiliar procedure · contested · reviewed by AI
- A large share of an agent benchmark score is attributable to the harness around the model rather than the model itself: holding the model fixed and changing only guides, sensors, retries and state management is reported to move the same model from 30.9% to 74.6% on GAIA and 30th to 5th on Terminal Bench — so a score reported for "a model" on an agentic benchmark is a score for a model-plus-harness, and comparisons across labs with different harnesses are not comparisons of models.Whether the measurement made the finding · reviewed by AI
- When a model gets a real environment signal after acting — a test result, a tool error, a task-success indicator — reflecting on that signal in words and retrying substantially improves success rates on multi-step coding and decision-making tasks over a single attempt.Fixing its own mistakes
- Giving an agent a parsed code graph (files, classes, functions with import, call and inheritance edges) and tools to search and traverse it lets a fine-tuned 32B open model localize issues about as well as proprietary agents at roughly an eighth of the cost, reaching about 93% file-level accuracy, and better localization carried through to about 12 points more resolved issues at Pass@10 when plugged into a fixed repair pipeline.Finding what matters in a large codebase · reviewed by AI
Notes
The SWE-agent brief carries one flagged figure, 2,294, which is the size of full SWE-bench and does not appear in the paper's text as fetched; the claim above does not use it.