code-generation CAP-0004 · activeGenerating and editing working code
Whether a model can write correct code from a specification, and go beyond isolated functions to real repository-scale editing and debugging.
Tags: reasoning, agentic
Distinct from secure-coding, which is specifically about vulnerability patterns and dependency hallucination. This is about raw functional correctness — does the generated code actually pass held-out tests — and, at the harder end, whether a model can navigate and edit an existing codebase rather than writing an isolated function from scratch. Static, fixed-problem benchmarks saturate and leak into training data over time, which is why contamination-resistant, continuously-updated benchmarks matter here more than in most areas.
Claims
- observationsingle paper↑1↓0For a 7B code model fine-tuned on competitive-programming tasks, training it to emit a fixed-schema requirement specification (goals, input/output constraints, examples, edge cases, dependencies) before the code raises Pass@1 more than training it to emit free-form reasoning or a stepwise implementation plan of similar length, with the largest margin on an out-of-distribution benchmark.
- mechanismsingle paper↑1↓0When an agent builds a whole Python repository from natural-language requirements alone, refining component boundaries with explicit cohesion and coupling thresholds until no split or merge is triggered produces higher functionality coverage and test pass rate than letting the LLM decide restructuring actions for a fixed budget of rounds, because unconstrained LLM-decided actions keep over-decomposing past the point of a coherent modular structure.
- observationsingle paper↑1↓0When an agent generates a whole Python repository from natural-language requirements with no given architecture, keeping the repository architecture as a mutable state that is re-evolved before code generation yields higher functionality coverage and test pass rate than executing a single up-front planning graph, and removing the evolution step is the largest single ablation loss.
- mechanismsingle paper↑1↓0In agent pipelines that generate whole domain-specific ML codebases (medical imaging) from scratch, execution/assembly validation and domain knowledge bases fix different failures: removing runtime and assembly testing mainly costs autonomy (more human interventions to debug), while removing the curated domain knowledge base mainly costs output quality (syntactically fine but domain-inappropriate pipelines) with little change in intervention count.
- mechanismsingle paper↑1↓0For automated unit test generation on domain-specific Java libraries (NLP toolkits), injecting LLM-written test snippets into a search-based generator's population when the evolutionary search stagnates yields higher branch and line coverage than either the search tool alone or the LLM alone, with the LLM contributing object-construction sequences and domain-shaped string inputs that random search does not reach.
- observationsingle paper↑1↓0When a code-generation prompt combines a prompt-enforced output format (JSON or XML), a persona, and urgency framing at once, pass@1 on function-level Python problems can fall well below the sum of each constraint's individual effect, even when each constraint alone is neutral or helpful — observed in the GPT-4o family and absent in the GPT-4.1 family and o3-mini.
- observationsingle paper↑1↓0For data-science Python tasks where several correct solutions differ by orders of magnitude in runtime, a model's correctness rate does not predict the efficiency of the code it writes: rankings invert between the two, code-specialized open models are less efficient than general open models of similar size class, and the fastest model rotates by library (NumPy vs Pandas vs PyTorch vs Polars).
- mechanismsingle paper↑1↓0For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.
- mechanismsingle paper↑1↓0When fine-tuning a 4B code model, training on auxiliary targets that name each program region's correctness obligations (derived from aligning an AST implementation graph with Lean proof-flow traces) raises solved counts over code-only SFT, but the auxiliary-target stage alone shifts the model off the Python-only output distribution and must be followed by a code-only consolidation stage to avoid losing broad function-level ability.
- observationsingle paper↑1↓0When fine-tuning 7-8B code models on an LLM-synthesized Verilog instruction corpus, training on a 20-25% subset selected for functional correctness, instruction-solution alignment, and coverage of Verilog structural clusters (AST, CFG, netlist features) yields higher Pass@1 than training on the full corpus, because the automated corpus carries noise and redundancy that filtering removes.
- observationsingle paper↑1↓0For a fixed 7B code backbone on short Python tasks (MBPP), task-aware prompt routing plus LoRA fine-tuning on prompts rewritten into the routed style raises pass@1 only slightly (+1.1 points) when one sample is returned, but yields large gains (+9-10 points) once several candidates are sampled and selected by execution against the task's visible public tests.
- mechanismsingle paper↑1↓0Grading generated code by whether it actually passes held-out functional tests, rather than by surface similarity to a reference solution, is what makes code generation gradeable at scale — and it's how the field has graded it since the earliest large benchmarks.
- mechanismreplicated↑1↓0For LLM-generated HLS kernels targeted at real FPGA boards, the two stages of failure respond to different interventions: an evolutionary search over simulation-validated kernel candidates mainly raises synthesizable and executable rates, while a C/C++-level functional verify-and-repair loop mainly raises functional correctness and barely moves synthesizability, so both are needed for end-to-end success.
- mechanismsingle paper↑1↓0For LLM-generated HLS kernels targeting real FPGA boards, simulation and synthesis success overstates deployability, and the two stages are fixed by different interventions: an evolutionary search over simulated candidates raises synthesizable and executable rates while barely raising functional correctness, whereas C/C++-level verify-and-repair raises functional correctness while leaving synthesizability flat.
- observationsingle paper↑1↓0On repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.
- observationsingle paper↑1↓0On repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.
- mechanismsingle paper↑1↓0When reranking sampled Verilog candidates with both a generated-testbench execution signal and an LLM judge, letting the judge see the execution outcomes in its prompt lowers final Pass@1 relative to acquiring the two signals separately and combining them only as a weighted score at selection time, because the judge anchors on the test verdicts (approving all-pass candidates with uncovered logic errors, over-penalizing failures caused by bad testbenches) and stops contributing information the tests miss.
- mechanismsingle paper↑1↓0In search-based automated program repair of Simulink/Stateflow models, replacing hand-crafted domain-specific mutation operators with an LLM that proposes label rewrites from static component context lowers repair coverage under a fixed wall-clock budget, because unconstrained generative edits widen the candidate space while each candidate costs an expensive simulation and the generator gets no behavioural feedback about why a patch failed.
- observationsingle paper↑1↓0On simple HLS kernel generation tasks (85 designs from CHStone, MachSuite, Polybench, Rosetta, C2HLSC), giving a modest open-source model (gpt-oss-120b) a shell with a C++ compiler for self-verification and iterative refinement saturates the benchmark at pass@10 (100% through synthesis) and exceeds 90% at pass@1, so this class of task no longer separates models; repeated sampling helps the weaker model far more (+60 points synthesis pass rate at k=10 for gpt-oss-20b vs +11 for the 120b).
- mechanismsingle paper↑1↓0Static, fixed-problem coding benchmarks saturate and leak into training data over time, so getting an honest read on current code-generation ability requires a benchmark that continuously adds newly-published problems rather than reusing an older fixed set like HumanEval.
- mechanismsingle paper↑1↓0In UI-to-code generation, where a rendered screenshot gives genuine external feedback, iterative self-critique still drifts below the initial draft over ten rounds, because a code edit changes the rendering non-locally and a fix for one region breaks regions that were already faithful; restricting each round to one typed, scoped repair target and carrying past targets forward as an avoid-list reduces that drift.
Techniques
- Evolutionary kernel search plus C-level functional repair, tprocessA multi-agent pipeline that injects Vitis HLS optimization priors, evolves kernel candidates under simulation feedback, verifies and repairs functional correctness by de-HLSifying the kernel and running C++ test programs, then generates hos
- Evolve component boundaries by modularity metrics until convprocessMaintain a requirement-level and a component-level DAG with an alignment relation, and apply split/merge/revise/save actions triggered by cohesion and coupling thresholds until no action is eligible, then generate code against the converged
- Inject proof-graph correctness obligations, then consolidatetrainingParses reference Python into a region-level implementation graph, aligns those regions with kernel-checked Lean proof-flow transitions to generate region-level obligation text, trains the model jointly on code plus obligation/concept predic
- Inject self-refined LLM test snippets when search stallstoolingWhen a search-based test generator makes no fitness progress for N generations, prompt an LLM at class level with an iterative self-refinement loop, parse its test snippets into the generator's internal representation, and merge them into t
- Mine training data through the language's own interpretertrainingGenerate candidate programs from natural-language instructions with a frontier LLM, keep only those that pass a deterministic-wording filter, compile and run in the target language's binary, produce non-trivial output, and pass an independe
- requirements-refinement-before-codingprocessrequirements-refinement-before-coding, as introduced by the paper it was drafted from.
- Route prompts, align LoRA to them, verify on public testsprocessPicks a task-specific prompt template by keyword heuristics, fine-tunes a LoRA adapter on training inputs rewritten into that same template style, then samples several programs and returns the one passing the most visible public tests.
- Select training data by quality and structural diversitytrainingFilter Verilog training samples by testbench-verified correctness and instruction-following-difficulty below 1, then cluster on fused text plus 109-dimensional AST/CFG/netlist features and sample proportionally per cluster by IFD rank.
- Validate HLS generation to board execution, not simulationprocessA multi-agent loop that injects Vitis pragma documentation, evolves kernel candidates under simulation-performance selection, verifies functional correctness on a de-pragma'd C++ harness with targeted repair, then generates host code and co
Sources
Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.
- ↑ supports a claim here
- Quality over Quantity: Diversity-Aware Data Selection for Efficient Verilog Code Generation2026-09-19↑ supports a claim here
- ↑ supports a claim here
- about this capability · no claim drafted yet
- about this capability · no claim drafted yet
- ↑ supports a claim here
- about this capability · no claim drafted yet
- SpecCoder: Specification-Aware Code Generation with Curriculum Dual-Task Reinforcement Learning2026-09-05↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here · ↑ supports a claim here
- ↑ supports a claim here
- Towards Fully Automated Medical Imaging Code Generation via Validation-based Context Engineering2026-08-29↑ supports a claim here
- about this capability · no claim drafted yet
- about this capability · no claim drafted yet
- XREPOTEST: Benchmarking Multilingual Repository-Level Unit Test Generation for Large Language Models2026-08-26about this capability · no claim drafted yet
- ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here · ↑ supports a claim here
- ↑ supports a claim here · ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- Pseudo2CodeQA: A Benchmark for LLM-Based Structured Algorithmic Reasoning in Code Generation2026-08-10about this capability · no claim drafted yet
- about this capability · no claim drafted yet
- ↑ supports a claim here
- Evaluating Large Language Models Trained on CodeThis source: Evaluating Large Language Models Trained on CodeHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 11293 totalpublished 2021checked 2026-09-04count capped at 1000 by the fetchclick for Semantic Scholar2021↑ supports a claim here
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for CodeThis source: LiveCodeBench: Holistic and Contamination Free Evaluation of Large Lan…How much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 2073 totalpublished 2024checked 2026-09-04count capped at 1000 by the fetchclick for Semantic Scholar2024↑ supports a claim here
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.