agent-builds-whole-python-repository-natural-language-requirements-alone-refining CLM-0357When an agent builds a whole Python repository from natural-language requirements alone, refining component boundaries with explicit cohesion and coupling thresholds until no split or merge is triggered produces higher functionality coverage and test pass rate than letting the LLM decide restructuring actions for a fixed budget of rounds, because unconstrained LLM-decided actions keep over-decomposing past the point of a coherent modular structure.
Capability: Generating and editing working code
Observed on
Six Python repositories in RepoCraft, generated from revised high-level requirement descriptions, with GPT-5 mini and DeepSeek V3.2 backbones and a shared downstream TDD generation.
Sources
- Validated by: Functionality Coverage, Pass Rate and Voting Rate on RepoCraft against mini-SWE-agent, Paper2Code and RPG with an identical downstream code-generation pipeline; reported gains over RPG of 4.55-20.08 points coverage and 7.61-29.74 points pass rate, plus ablations removing structural evolution and a comparison with LLM-decided evolution at 1, 3 and 5 rounds.. Measured on six repositories with two backbones, three runs averaged, cross-model evaluation; main table reports only three repositories. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When an agent generates a whole Python repository from natural-language requirements with no given architecture, keeping the repository architecture as a mutable state that is re-evolved before code generation yields higher functionality coverage and test pass rate than executing a single up-front planning graph, and removing the evolution step is the largest single ablation loss.Generating and editing working code · reviewed by AI
- For LLM agents that must compose and then execute domain tool workflows, training separate adapters for the two roles splits the gains cleanly: supervised fine-tuning of the planner on reference tool sequences mainly improves workflow-order fidelity, while node-level RL on argument-match rewards mainly improves tool-argument grounding and final-answer quality, and combining both beats either on a 9B backbone.Planning a sequence of actions before acting · reviewed by AI
- For LLM agents under a fixed ReAct harness answering questions over executable database-backed APIs plus document retrieval, accuracy falls sharply as the required reasoning depth grows from 1 to 5 hops (most models lose over half their accuracy), and the surviving failures concentrate in language-mediated steps — entity disambiguation, cross-source grounding, schema alignment — rather than in emitting syntactically valid tool calls.Carrying a chain of inference through to its conclusion · reviewed by AI
- When a small (1.7B) LLM is fine-tuned to act as an ensemble router that assigns per-sample weights across candidate time-series forecasters, chain-of-thought supervision built by rule-templating known oracle weights against extracted temporal features matches or beats CoT generated by a frontier API model, and removing the CoT step costs far more on unseen candidate model pools than on in-distribution ones.Predicting future events · reviewed by AI
- On repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.Generating and editing working code · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: On the same benchmark and scaffold, LLM-decided structural evolution with more rounds matching or beating the metric-thresholded version on coverage and pass rate, or removing the structural-evolution loop not reducing those scores.