arxiv-2310-06770 · paper

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Carlos E. Jimenez, John Yang, Alexander Wettig, et al.

Created: 2023 · Ingested: 2026-09-02

This source: SWE-bench: Can Language Models Resolve Real-World GitHub Issues?How much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 3594 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetch1000+ citations in the last 12 months · 3594 total · checked 2026-09-04

https://arxiv.org/abs/2310.06770(opens in a new tab)

In brief

Unverified figures: 19000. These do not appear in the abstract this was written from, so read the paper before quoting them.

Frontier language models in 2023 could not resolve real GitHub issues: the best system fixed under 2% of them when it had to find its own context.

SWE-bench pairs 2294 merged pull requests from 12 popular Python repositories with the issues they closed and the tests they added. A model sees the issue text plus retrieved files from a codebase averaging 3,010 non-test files and 438K lines, and must emit a patch that turns at least 1 failing test green without breaking a median of 51 other tests. Evaluated: ChatGPT-3.5, GPT-4, Claude 2, Claude 3 Opus, and CodeLlama 7b/13b fine-tuned on 19000 issue-PR pairs from 37 disjoint repositories (SWE-Llama).

With BM25 retrieval, Claude 2 resolves 1.97%, GPT-4-turbo 1.31%, ChatGPT-3.5 0.17%, SWE-Llama 13b 0.70%. Giving models the oracle files raises Claude 2 to 4.8%; collapsing those files to the edited lines ±15 raises it to 5.93% and GPT-4 from 1.3% to 3.4%. Asking for whole files instead of patches drops Claude 2 to 2.2%.

The retrieval and format ablations isolate localization and output format as separate bottlenecks. Pre/post-2023 splits show little date effect, arguing against contamination, though GPT-4 was run on a 25% subset. Python only, single-turn prompting, no agent scaffolds tested.

Treat the low numbers as a floor for non-agentic setups, not a ceiling on the task.

Written from the abstract by claude-opus-5 on 2026-09-05, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

No claims cite this yet.