fine-tuning-7-8b-code-models-llm-synthesized-verilog-instruction-corpus-training CLM-0391observationsingle paperReviewed by AI
When fine-tuning 7-8B code models on an LLM-synthesized Verilog instruction corpus, training on a 20-25% subset selected for functional correctness, instruction-solution alignment, and coverage of Verilog structural clusters (AST, CFG, netlist features) yields higher Pass@1 than training on the full corpus, because the automated corpus carries noise and redundancy that filtering removes.
Capability: Generating and editing working code
Observed on
RTLCoder-27k as the training pool; Qwen2.5-Coder 7B, CodeLlama 7B, SeedCoder 8B with LoRA SFT; VerilogEval-v2, RTLLM, ResBench as tests.
Sources
- Validated by: Average Pass@1 across three benchmarks versus full-dataset training and versus Random, EL2N, Cherry, VeriCoder and a matched-budget random subsample of VeriCoder's filtered pool; 18.49-29.43% relative gain over full data, 1.36-7.95% over best baseline; McNemar p<0.01 against Cherry on SeedCoder 8B; ablations per feature group.. Measured on three models and three benchmarks with 20 samples per problem, including a matched-budget controlled baseline (VeriCoder_R) and component ablations; [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Generating and editing working code · reviewed by AI
- For a same-family pair of reasoning and instruction-tuned 4B models, learning one interpolation ratio per module (attention block, FFN block, embedding, final norm) with a reward that scores correctness and ranks correct responses by length cuts average generated tokens by about 27% relative to the reasoning model while holding average accuracy, and beats fixed global-ratio merging and CMA-ES search over the same ratios.Cutting to the chase · reviewed by AI
- For instruction tuning an 8B open model on text-to-SQL, a curated set of under 900 examples selected to cover every table in the target schemas and score well on question-SQL alignment reaches higher execution accuracy than pipelines trained on tens of thousands to millions of samples, and dropping the schema-coverage term from the selection objective costs more accuracy than dropping any single quality term.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Training the same models on the full RTLCoder-27k corpus matching or beating the selected subset on Pass@1, or the subset advantage vanishing once the matched-budget random-subsample baseline is tuned or epochs equalized by compute.