domain-specific-language-essentially-absent-pretraining-corpora-form-symbolic-algebra CLM-0377For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.
Capability: Generating and editing working code
Observed on
Zero-resource DSL with a deterministic execution oracle; single-attempt, temperature-0 generation; frontier baselines are open-weight only and get 6,000 characters of docs; benchma.
Sources
- Validated by: 840 single-attempt tasks across four benchmarks: 97.7% exact output match on 664 deterministic tasks vs 75.5% for GLM-5.2 and 62.3% for DeepSeek-V4 Flash (non-overlapping 95% CIs); 83.0% execution and 18.0% strict match on Instruct-100 vs 65.0%/5.6% best frontier (McNemar p<=0.007); no lead on the 41- and 22-task benchmarks (p>=0.375); MMLU -2.6pp, GSM8K -1.5pp, HumanEval -0.6pp.. Measured, single author, single base model, author-built benchmarks. Verified by execution against the FORM binary with difference-to-zero equality, which is strong. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Adding a repository-wide line-level graph of definitions and references, queried for the ego-graph around names the model is looking at, raised SWE-bench resolve rates for all four frameworks it was plugged into (two agentic, two procedural) by an average of about 33% relative, at a modest cost increase that was larger for the agentic ones; the same plug-in also helped on CrossCodeEval.Finding what matters in a large codebase · reviewed by AI
- When instruction-tuned models (Qwen2.5 7B-32B) make function calls from non-English user requests, they often select the right tool and argument names but emit argument values in the wrong language, a failure that exact-match or AST tool-calling metrics record as a generic invocation error rather than a language mismatch.Using the tools it is given · reviewed by AI
- When fine-tuning 7-8B code models on an LLM-synthesized Verilog instruction corpus, training on a 20-25% subset selected for functional correctness, instruction-solution alignment, and coverage of Verilog structural clusters (AST, CFG, netlist features) yields higher Pass@1 than training on the full corpus, because the automated corpus carries noise and redundancy that filtering removes.Generating and editing working code · reviewed by AI
- For small open-weight Qwen2.5 models (1.5B-3B) doing multi-step math reasoning, features derived from the spread of attention entropy across layers at each generation step predict whether the final answer is wrong better than output-logit confidence features, with the margin larger on harder problems (+0.0093 AUC on GSM8K, +0.0759 on MATH-500 at 3B).Stating false facts confidently · reviewed by AI
- On long-context tasks over 8K-128K-token documents in the six official UN languages, models answer word-level span questions well (about 73% average accuracy across 12 models) but degrade sharply as the required unit coarsens to sentence selection (about 21%), paragraph filling and document-level summarization and translation, so long-context competence measured by needle-style word retrieval overstates comprehension of larger discourse units.Losing information in long inputs · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A frontier model with the same documentation matching or beating the fine-tuned 8B specialist on the 664-task deterministic and Instruct-100 strict metrics, or the specialist's lead disappearing on a benchmark built by an independent party.