grammar-constrained-text-to-sql-decoding-small-models-qwen2-5-instruct-0-5b-7b CLM-0400For grammar-constrained text-to-SQL decoding with small models (Qwen2.5-Instruct 0.5B-7B on Spider), beam search is at least as accurate as sample-and-vote at every matched compute budget of 1-8, and better in most, reversing the usual unconstrained finding that sampling with voting beats beam search.
Capability: Writing a reusable locator: regex, SQL, XPath
Observed on
One model family (Qwen2.5-Instruct, 4-bit), one benchmark (Spider dev, n=1034), schema-aware CFG constraint, sample+vote at temperature 0.7, single decoding run per configuration..
Sources
- Validated by: Execution accuracy on the full 1,034-example Spider dev set at budgets 1, 2, 4, 8 for four model sizes; paired McNemar exact tests on the 16 matched-budget cells — beam significantly better in 11, significantly worse in 0, ties in 5.. Measured, not argued: one decoding run per cell, single seed for sampling, paired McNemar on identical examples so the comparison is properly paired. Confounded by the specific grammar, which the authors show is incomplete (it breaks more predictions than it fixes at 3B and 7B), and by a fixed temperature of 0.7 with no sweep. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On text-to-SQL benchmarks, aggregate accuracy hides a steep difficulty gradient: pre-trained-language-model-based parsers resolved roughly 80% of Spider cases overall but only about 50% of its extra-hard cases, and as natural-language understanding improved the residual errors shifted toward database-side problems — large schemas, ambiguous columns, dirty cell values — rather than language parsing.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- When fine-tuning 7-8B code models on an LLM-synthesized Verilog instruction corpus, training on a 20-25% subset selected for functional correctness, instruction-solution alignment, and coverage of Verilog structural clusters (AST, CFG, netlist features) yields higher Pass@1 than training on the full corpus, because the automated corpus carries noise and redundancy that filtering removes.Generating and editing working code · reviewed by AI
- For instruction tuning an 8B open model on text-to-SQL, a curated set of under 900 examples selected to cover every table in the target schemas and score well on question-SQL alignment reaches higher execution accuracy than pipelines trained on tens of thousands to millions of samples, and dropping the schema-coverage term from the selection objective costs more accuracy than dropping any single quality term.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- On text-to-SQL questions that are underspecified with respect to the schema, retrieving clause-level fragments (conditions, joins, calculations, groupings, output specs) from a database's own past successful SQL queries and prepending them to existing few-shot pipelines raises execution accuracy across four different pipelines, but the gain saturates and can reverse once too many candidates are retrieved, because high recall with low precision lets the generator pick a plausible-but-wrong pattern.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Generating and editing working code · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: On another model family, constraint type (JSON schema, tool-call grammar), or benchmark, sample+vote at a matched budget beats beam search under grammar-constrained decoding; or repeated seeds on Spider show beam's advantage inside sampling noise.