swe-bench-lite-asking-model-name-suspicious-files-issue-plus CLM-0271observationsingle paperReviewed by AI
On SWE-bench Lite, asking the model to name suspicious files from the issue plus a repository tree finds the ground-truth file in about 79% of cases and embedding retrieval over the issue text in about 68%, and taking the union of the two raises file-level localization to about 82% because the two methods miss different files; a fixed three-step narrowing (files, then classes and functions, then edit lines) with no agent loop then supported the best open-source resolve rate at the time (32%).
Capability: Finding what matters in a large codebase
Observed on
SWE-bench Lite (300 Python issues), GPT-4o, 2024; localization measured by whether the ground-truth edit locations remain in the candidate set after each step..
Sources
- Validated by: Contains-ground-truth rate after each localization step on SWE-bench Lite with GPT-4o (file-level 78.7% prompting, 67.7% embedding, 81.7% combined), and end-to-end resolve rate and cost against contemporaneous open-source agents; no comparison against a tool-using agent given the same localization.. Numbers are for one model on one benchmark and the paper's motivation is to argue against agents, but the localization ablation is direct and per step. Python only; SWE-bench Lite has known leakage and underspecification the paper itself documents.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Giving an agent a parsed code graph (files, classes, functions with import, call and inheritance edges) and tools to search and traverse it lets a fine-tuned 32B open model localize issues about as well as proprietary agents at roughly an eighth of the cost, reaching about 93% file-level accuracy, and better localization carried through to about 12 points more resolved issues at Pass@10 when plugged into a fixed repair pipeline.Finding what matters in a large codebase · reviewed by AI
- Constraining a localization agent with a priority queue over candidate code regions scored for relevance, decomposition of each search action, and pruning of context by distance from the issue's anchors reached a 65% function-level match rate on SWE-bench Lite, the best open-source result at the time, and feeding its locations to an open-source repair pipeline raised the resolved rate by about 6 points.Finding what matters in a large codebase · reviewed by AI
- Without training or a prebuilt index, searching a module call graph built on the fly for breadth at file level and then expanding it into a function call graph for depth reached about 43-44% function-level Top-1 localization on SWE-bench Lite and Verified with open 32B and 72B models, beating prior LLM localizers, and the better locations raised an issue-solving pipeline's resolve rate by roughly 3 to 30 points depending on model.Finding what matters in a large codebase · reviewed by AI
- Issue localization can be done as retrieval-and-rerank with no agent at all: an embedding model and reranker trained on issue-to-patched-function pairs mined from public GitHub (SweLoc) localize functions on SWE-bench Lite and LocBench at least as accurately at Acc@10 as agents built on Claude 3.5 Sonnet, at a small fraction of the per-issue cost and with one pass instead of seven to ten model rounds.Finding what matters in a large codebase · reviewed by AI
- Adding a repository-wide line-level graph of definitions and references, queried for the ego-graph around names the model is looking at, raised SWE-bench resolve rates for all four frameworks it was plugged into (two agentic, two procedural) by an average of about 33% relative, at a modest cost increase that was larger for the agentic ones; the same plug-in also helped on CrossCodeEval.Finding what matters in a large codebase · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A repository or language where the file tree is too large or uninformative to rank from, so that tree-based prompting falls well below embedding retrieval, or an agent loop that localizes better at similar cost.