debugging CAP-0008 · activeFinding and fixing a fault in existing code
Whether a model can localise the cause of a failing behaviour in code it did not just write and repair it without breaking something else; distinct from generating code and from self-repair of its own output.
Tags: coding-agent, agentic
Filed from Russ's capability map (2026-09-12). Fixing its own mistakes covers a model correcting its own fresh output with or without feedback. Debugging is the harder, more common case in practice: a fault in an unfamiliar codebase, a failing test, a bug report. Repository-level issue-fixing benchmarks measure this together with repository navigation.
What counts as this capability
Scope boundary used when deciding whether a paper is really about this capability, rather than merely mentioning it.
In scope when the paper measures fault localisation or repair on code the model did not author in the same session. NOT in scope for self-repair of the model's own draft, or for code generation from a specification.
Claims
- mechanismsingle paper↑1↓0Every state-of-the-art model tested repairs fewer Defects4J and GitBug-Java bugs after semantics-preserving transformation of the buggy code, from -4% for GPT-4o to -16% for Llama-3.1, and the size of each drop correlates with how likely the model was to have memorized the original (negative log-likelihood), so part of published repair success on these benchmarks is recall of seen fixes rather than debugging.
- observationsingle paper↑1↓0Fine-tuned program-repair models that score well on HumanEval-Java-Bug lose more than half their Pass@10 under semantics-preserving rewrites of the buggy code, with local-variable renaming alone cutting performance by 50-57%, so much of measured repair skill is tied to the exact surface form of the bug.
- observationsingle paper↑1↓0On real C/C++ faults with reproducing tests, 24 current models repair only about 11% of the 248 bugs and about 6% of the 102 vulnerabilities, far below their Java and Python repair rates on comparable benchmarks, and fine-tuning on the benchmark's commit corpus raised Pass@k in 21 of 28 comparisons by an average 85% relative, so the gap is partly a training-data gap.
- observationsingle paper↑1↓0For repair of production crash bugs in large Java enterprise systems, feeding an LLM the source of the top-3 suspicious files plus a representative stack trace localizes the developer-modified code in about 71% of responses and yields a semantically valid patch in about 52%, and adding an explicit ranking of suspicious files and methods helps only when the full stack trace is withheld (per-response repair 28% to 47% with stack messages only) while slightly lowering performance when the stack trace is already present.
Techniques
- Rank crash-clustered suspicious files as LLM repair contextretrievalCluster crash reports by stack-trace similarity, score candidate files by proximity to the crash point, bucket specificity and in-group frequency, and put the top-ranked files' source plus a representative stack trace into a zero-shot repai
Sources
Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.
- about this capability · no claim drafted yet
- Integrating Crash Report Mining and LLMs for Bug Localization and Repair: An Industrial Report2026-09-03↑ supports a claim here
- about this capability · no claim drafted yet
- ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.