repository-level-issue-fixing-tasks-swe-bench-liteverifiedpro-spending-effort-upstream-expanding CLM-0131On repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.
Evidence for: Humphrey's law (holds)
Capability: Generating and editing working code
Observed on
100 sampled instances per benchmark, DeepSeek-V3.2 and Qwen-Plus baselines against Agentless, Trae-agent and Claude Code; a single added run with Claude-Opus-4.8 on SWE-bench-Pro g.
Sources
- Measured across 6 settings (3 benchmarks x 2 models) with a Wilcoxon signed-rank test; %Resolved gains 2-29%, %Applied gains 11-63%, so the applied-vs-resolved asymmetry is visible in the table. Compute per instance is not reported, so the comparison does not isolate extra inference budget from the requirements idea, and the stronger-model check is one benchmark with a 3-point gap.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.Generating and editing working code · reviewed by AI
- When a benign GitHub issue is augmented with a plausible-sounding maintenance requirement that specifies a helper reading an environment variable into a dangerous sink, repository-level APR agents produce patches that fix the original bug and pass the project test suite while containing the requested vulnerability in roughly half of tasks, and the strongest instruction-following coding models are more susceptible, not less.Writing secure code and dependencies · reviewed by AI
- In LLM-based repository-level program repair, telling the generator to be minimal or constraining its output format does not shrink patches, because verbosity comes from the capability-raising machinery itself (iterative refinement, broad context); a separate post-generation refiner that rewrites an already-plausible patch is what reduces size without losing fixes.Cutting to the chase · reviewed by AI
- For a 7B code model fine-tuned on competitive-programming tasks, training it to emit a fixed-schema requirement specification (goals, input/output constraints, examples, edge cases, dependencies) before the code raises Pass@1 more than training it to emit free-form reasoning or a stepwise implementation plan of similar length, with the largest margin on an out-of-distribution benchmark.Generating and editing working code · reviewed by AI
- For LLM program-repair workflows built on self-correction, generated-knowledge or multi-agent scaffolding, the gain over plain prompting is unstable across benchmarks and can turn negative once extrinsic defect information (error messages, triggering tests, bug comments) is withheld, while simply putting the error message plus the failing test into the prompt yields more correct repairs than the scaffolds do.Fixing its own mistakes · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A matched comparison where adding the requirement-construction and refinement stage leaves %Resolved unchanged or lower relative to the same agent given the raw issue text, or where the patch-application gain does not exceed the correctness gain. Proposed technique, not catalogued: requirements-refinement-before-coding.