file-based-sandbox-agent-tasks-distractor-artefacts-scoring-only-whether CLM-0567In file-based sandbox agent tasks with distractor artefacts, scoring only whether the task goal is met overstates reliability, because agents frequently complete the target while modifying unrelated files: across nine LLM agents, pass rate exceeds a success-plus-preservation metric by roughly 10-17 points, with the gap widening as model capability falls.
Capability: Completing tasks that take many steps or many hours
Observed on
1,600 persona-grounded file workflows (spreadsheets, documents, PDFs, messaging, calendar, shell) in seven languages, each testbed seeded with 20 distractor files; one run per mode.
Sources
- Validated by: Applied to nine agents; the combined metric (CTS) separated models differently from pass rate, with best model 49.2% CTS vs 59.3% pass, and collateral edits appearing as a substantial failure category for all agents.. Measured on one benchmark the authors built, nine models, single run each, deterministic plus LLM-judge evaluators. Preservation is checked by byte-identity, so benign reformatting counts as failure; the design does not separate harmful collateral edits from harmless ones. Language gradient and failure-mode shares are from the same single runs.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For LLM agents that score their own actions during exploration in text-based games and then reuse that experience, the accuracy of those self-scores against the environment verifier does not predict later gains on held-out stricter configurations: block-level correlations between judgment agreement or calibration and the next checkpoint's score change sit near zero, so recognizing a good action is not the bottleneck — converting it into a transferable policy is.Improving its own scaffolding from failures · reviewed by AI
- For single-agent procedural tasks where the relevant state fits a fixed schema known in advance, replacing the appended conversation transcript with a structured execution state that the model rewrites each step — discarding the reasoning trace once the state patch is validated — holds accuracy flat as the horizon grows while history-appending runtimes degrade, and the gain comes from the structure rather than the shorter prompt.Completing tasks that take many steps or many hours · reviewed by AI
- For LLM web agents on cluttered pages containing many concurrent interface components, success falls sharply as component count grows, and having the agent first emit a localized sub-goal plus a task-relevant sub-observation before choosing an action recovers much of the loss, because self-assessment of the action is more reliable in the reduced context.Using the tools it is given · reviewed by AI
- When an agent system evolves its own harness from failure traces, batching failures across tasks before editing buys efficiency but not better failure attribution: pooled across three agent benchmarks, the share of harness edits that merely accommodate the current model rose from 60.0% under per-failure serial editing to 75.6% under cross-task aggregation, and only fell to 45.5% when a separate analyst/critic/engineer refinement stage attributed each recurring pattern before the edit was written.Improving its own scaffolding from failures · reviewed by AI
- In LLM multi-agent systems that reflect after failure, restricting reflection to the single agent responsible for the first incorrect step — located by a trained step-level correctness scorer and paired with a counterfactually generated corrected step — raises task success more than having every agent reflect on the trajectory, because reflections written by agents that behaved correctly enter their memory as misleading guidance for later tasks.Getting more from several models than from one · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A replication on comparable file-based agent tasks where pass rate and preservation-constrained success are near-identical, i.e. agents almost never alter non-target files, or where the gap shrinks rather than grows for weaker models.