7b-code-model-fine-tuned-competitive-programming-tasks-training-emit-fixed-schema CLM-0195For a 7B code model fine-tuned on competitive-programming tasks, training it to emit a fixed-schema requirement specification (goals, input/output constraints, examples, edge cases, dependencies) before the code raises Pass@1 more than training it to emit free-form reasoning or a stepwise implementation plan of similar length, with the largest margin on an out-of-distribution benchmark.
Capability: Generating and editing working code
Observed on
Qwen2.5-7B-Coder-Instruct, specification-rich competitive programming benchmarks (APPS, CodeContests in-distribution; xCodeEval out-of-distribution); intermediate representations f.
Sources
- supportsSpecCoder: Specification-Aware Code Generation with Curriculum Dual-Task Reinforcement LearningOne backbone, one paper. Generation-only ablation (Table 5): APPS 0.172 vs 0.166/0.170/0.168, CodeContests 0.061 vs 0.055/0.060/0.061, xCodeEval 0.127 vs 0.080/0.103/0.093, with reported 95% CIs that overlap on the in-distribution benchmarks; only the OOD gap is wide relative to the intervals. Output length is reported (694.7 tokens vs 568.2/539.0), so the comparison is not length-matched, though the authors argue length alone does not explain the gap. No second backbone or seed variation.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For instruction tuning an 8B open model on text-to-SQL, a curated set of under 900 examples selected to cover every table in the target schemas and score well on question-SQL alignment reaches higher execution accuracy than pipelines trained on tens of thousands to millions of samples, and dropping the schema-coverage term from the selection objective costs more accuracy than dropping any single quality term.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- On repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.Generating and editing working code · reviewed by AI
- For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Generating and editing working code · reviewed by AI
- On repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.Generating and editing working code · reviewed by AI
- Extended reasoning budgets improve a model's execution of arithmetic procedures on operands longer than those seen in training far more than they improve surface-form-to-value tasks like magnitude comparison and format conversion, because chain-of-thought supplies external working memory for carries and column alignment but cannot change how the input was tokenized.Digit-level arithmetic · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafter linked technique "chain-of-thought", which does not list this capability in addresses -- recorded, not asserted. Drafted confidence: low. Falsifier as drafted: Under the same matched training budget and data pipeline, free-form rationale or planning-CoT pre-generation matching or exceeding the fixed-schema specification arm's Pass@1, especially on held-out out-of-distribution benchmarks. Proposed technique, not catalogued: Structured specification analysis before code.