boluk-2026-harness-problem-15-llms · post

I Improved 15 LLMs at Coding in One Afternoon. Only the Harness Changed.

Can Bölük

Created: 2026-02-12 · Ingested: 2026-09-11

https://can.ac/2026/02/12/the-harness-problem/

Security research, reverse engineering & low-level systems Snapcompact: SoTA Compaction — Instant, Local, Free. Pick 3 I Improved 15 LLMs at Coding in One Afternoon. Only the Harness Changed.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsGiving a coding agent a raw shell is worse than giving it a designed interface: SWE-agent's constrained Agent-Computer Interface — a file viewer windowed to 100 lines, a search tool that truncates large output, and an editor that lints before a change persists — roughly doubled SWE-bench resolution over raw shell access with the same model, because verbose output and unmanaged state are where the model loses its place.
supportsA large share of an agent benchmark score is attributable to the harness around the model rather than the model itself: holding the model fixed and changing only guides, sensors, retries and state management is reported to move the same model from 30.9% to 74.6% on GAIA and 30th to 5th on Terminal Bench — so a score reported for "a model" on an agentic benchmark is a score for a model-plus-harness, and comparisons across labs with different harnesses are not comparisons of models.