post-training-7-8b-open-model-text-to-sql-fixed-budget-synthetic-examples CLM-0625observationsingle paperReviewed by AI
When post-training a 7-8B open model for text-to-SQL with a fixed budget of synthetic examples drawn from a large external pool, selecting examples whose schema-independent SQL skeletons are farthest from those already in the seed set (greedy K-center) rather than sampling randomly from the same pool yields a small in-domain gain and a larger cross-domain one (BIRD dev 64.9 vs 64.4, Spider Test 85.3 vs 83.4, Science Benchmark 53.9 vs 49.8).
Capability: Writing a reusable locator: regex, SQL, XPath
Observed on
Qwen2.5-Coder-7B-Instruct and Llama-3.1-8B-Instruct post-trained with GRPO plus failure-driven SFT; 3,000 BIRD seed examples plus 3,500 synthesized examples instantiated on BIRD da.
Sources
- supportsCOAL-SQL: Coverage-Guided Augmentation and Failure-Driven Learning for Text-to-SQL Post-TrainingValidated by: Execution accuracy on BIRD dev, Spider test/dev, EHRSQL and Science Benchmark against random augmentation from the same pool at equal example count, plus skeleton n-gram coverage/recovery on BIRD dev versus OmniSQL-style synthesis (100% vs 73.3% unigram recovery).. Single ablation row per configuration, one backbone for the ablation, no seeds or variance reported; the augmentation arm is confounded with the surrounding failure-driven learning stages, though those are held fixed. [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For instruction tuning an 8B open model on text-to-SQL, a curated set of under 900 examples selected to cover every table in the target schemas and score well on question-SQL alignment reaches higher execution accuracy than pipelines trained on tens of thousands to millions of samples, and dropping the schema-coverage term from the selection objective costs more accuracy than dropping any single quality term.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- On text-to-SQL questions that are underspecified with respect to the schema, retrieving clause-level fragments (conditions, joins, calculations, groupings, output specs) from a database's own past successful SQL queries and prepending them to existing few-shot pipelines raises execution accuracy across four different pipelines, but the gain saturates and can reverse once too many candidates are retrieved, because high recall with low precision lets the generator pick a plausible-but-wrong pattern.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- For selecting among multiple generated SQL candidates, a listwise selector given retrieved structured selection criteria distilled from training question-SQL pairs, with candidates grouped by execution result and permuted only within groups, reaches higher execution accuracy than pointwise, pairwise and multi-selector baselines on the same candidate pools while using far fewer selector calls and tokens.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- On natural-language-to-SQL synthesis over multi-table databases, a model's accuracy overstates its ability whenever the same database or the same query appears in both training and test: splitting by database rather than by example cut the best system's exact-match accuracy from 33.0% to 12.4%, and accuracy falls further as the target database has more foreign keys.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- On text-to-SQL benchmarks, aggregate accuracy hides a steep difficulty gradient: pre-trained-language-model-based parsers resolved roughly 80% of Spider cases overall but only about 50% of its extra-hard cases, and as natural-language understanding improved the residual errors shifted toward database-side problems — large schemas, ambiguous columns, dirty cell values — rather than language parsing.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: Running the same pipeline with randomly sampled pool examples at equal budget and matching or exceeding coverage-guided selection on out-of-domain benchmarks, across more than one seed and backbone.