repository-level-issue-fixing-tasks-swe-bench-lite-verified-pro-spending-effort CLM-0244On repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.
Capability: Generating and editing working code
Observed on
100 sampled instances per benchmark, DeepSeek-V3.2 and Qwen-Plus baselines against Agentless, Trae-agent and Claude Code; a single added run with Claude-Opus-4.8 on SWE-bench-Pro g.
Sources
- Validated by: Not stated. Measured across 6 settings (3 benchmarks x 2 models) with a Wilcoxon signed-rank test; %Resolved gains 2-29%, %Applied gains 11-63%, so the applied-vs-resolved asymmetry is visible in the table. Compute per instance is not reported, so the comparison does not isolate extra inference budget from the requirements idea, and the stronger-model check is one benchmark with a 3-point gap.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.Generating and editing working code · reviewed by AI
- When a benign GitHub issue is augmented with a plausible-sounding maintenance requirement that specifies a helper reading an environment variable into a dangerous sink, repository-level APR agents produce patches that fix the original bug and pass the project test suite while containing the requested vulnerability in roughly half of tasks, and the strongest instruction-following coding models are more susceptible, not less.Writing secure code and dependencies · reviewed by AI
- In LLM-based repository-level program repair, telling the generator to be minimal or constraining its output format does not shrink patches, because verbosity comes from the capability-raising machinery itself (iterative refinement, broad context); a separate post-generation refiner that rewrites an already-plausible patch is what reduces size without losing fixes.Cutting to the chase · reviewed by AI
- When an agent builds a whole Python repository from natural-language requirements alone, refining component boundaries with explicit cohesion and coupling thresholds until no split or merge is triggered produces higher functionality coverage and test pass rate than letting the LLM decide restructuring actions for a fixed budget of rounds, because unconstrained LLM-decided actions keep over-decomposing past the point of a coherent modular structure.Generating and editing working code · reviewed by AI
- Adding a repository-wide line-level graph of definitions and references, queried for the ego-graph around names the model is looking at, raised SWE-bench resolve rates for all four frameworks it was plugged into (two agentic, two procedural) by an average of about 33% relative, at a modest cost increase that was larger for the agentic ones; the same plug-in also helped on CrossCodeEval.Finding what matters in a large codebase · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A matched comparison where adding the requirement-construction and refinement stage leaves %Resolved unchanged or lower relative to the same agent given the raw issue text, or where the patch-application gain does not exceed the correctness gain.