repository-level-issue-fixing-tasks-swe-bench-liteverifiedpro-spending-effort-upstream-expandingOn repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Generating and editing working code
Observed on
100 sampled instances per benchmark, DeepSeek-V3.2 and Qwen-Plus baselines against Agentless, Trae-agent and Claude Code; a single added run with Claude-Opus-4.8 on SWE-bench-Pro g.
Sources
- Measured across 6 settings (3 benchmarks x 2 models) with a Wilcoxon signed-rank test; %Resolved gains 2-29%, %Applied gains 11-63%, so the applied-vs-resolved asymmetry is visible in the table. Compute per instance is not reported, so the comparison does not isolate extra inference budget from the requirements idea, and the stronger-model check is one benchmark with a 3-point gap.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- For locally deployed open-weight models driving a stateful, dependency-ordered MCP tool server, cutting tool descriptions from full specifications (purpose, parameter semantics, constraints, failure conditions) down to one sentence each raises the fraction of calls the server rejects for every model tested, while its effect on task coverage is less consistent.Using the tools it is given · unreviewed
- On code prompts that are unsatisfiable by design, open-weight code and reasoning models refuse based on how suspicious the request looks rather than how deeply impossible it is: they refuse famous theoretical impossibilities far more often than requests for plausible-sounding nonexistent packages, and the stronger models in the set close the gap on theory while barely improving on fabricated ecosystem entities.Stating false facts confidently · unreviewed
- When grading LLM-generated unit tests by pass rate and line coverage alone, a share of passing test suites never directly call the function under test — they reach it through wrappers, mocks, or unrelated public APIs — so pass rate overstates test quality, and this gap widens under agentic repository-access workflows and richer file-level context.Whether the measurement made the finding · unreviewed
- In agent pipelines that generate whole domain-specific ML codebases (medical imaging) from scratch, execution/assembly validation and domain knowledge bases fix different failures: removing runtime and assembly testing mainly costs autonomy (more human interventions to debug), while removing the curated domain knowledge base mainly costs output quality (syntactically fine but domain-inappropriate pipelines) with little change in intervention count.Generating and editing working code · unreviewed
- Static, fixed-problem coding benchmarks saturate and leak into training data over time, so getting an honest read on current code-generation ability requires a benchmark that continuously adds newly-published problems rather than reusing an older fixed set like HumanEval.Generating and editing working code
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A matched comparison where adding the requirement-construction and refinement stage leaves %Resolved unchanged or lower relative to the same agent given the raw issue text, or where the patch-application gain does not exceed the correctness gain. Proposed technique, not catalogued: requirements-refinement-before-coding.