natural-language-to-sql-synthesis-over-multi-table-databases-model-s-accuracy-overstates CLM-0268On natural-language-to-SQL synthesis over multi-table databases, a model's accuracy overstates its ability whenever the same database or the same query appears in both training and test: splitting by database rather than by example cut the best system's exact-match accuracy from 33.0% to 12.4%, and accuracy falls further as the target database has more foreign keys.
Capability: Writing a reusable locator: regex, SQL, XPath
Observed on
Complex cross-domain text-to-SQL (Spider, 200 databases, 138 domains); 2018-era neural semantic parsers (Seq2Seq variants, SQLNet, TypeSQL), value prediction excluded..
Sources
- Validated by: Exact-match and per-component F1 for five models under an example split versus a database split of the same corpus; every model scored far lower under the database split, with TypeSQL dropping most (33.0 to 8.2 on test).. Measured on a 10,181-question, 5,693-query human-labeled corpus with five contemporary models under two splits; the split comparison isolates generalization to unseen schemas, but all models are pre-LLM so the absolute numbers are stale and the foreign-key trend is a plot without controls for query difficulty.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For instruction tuning an 8B open model on text-to-SQL, a curated set of under 900 examples selected to cover every table in the target schemas and score well on question-SQL alignment reaches higher execution accuracy than pipelines trained on tens of thousands to millions of samples, and dropping the schema-coverage term from the selection objective costs more accuracy than dropping any single quality term.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- On text-to-SQL benchmarks, aggregate accuracy hides a steep difficulty gradient: pre-trained-language-model-based parsers resolved roughly 80% of Spider cases overall but only about 50% of its extra-hard cases, and as natural-language understanding improved the residual errors shifted toward database-side problems — large schemas, ambiguous columns, dirty cell values — rather than language parsing.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- On text-to-SQL questions that are underspecified with respect to the schema, retrieving clause-level fragments (conditions, joins, calculations, groupings, output specs) from a database's own past successful SQL queries and prepending them to existing few-shot pipelines raises execution accuracy across four different pipelines, but the gain saturates and can reverse once too many candidates are retrieved, because high recall with low precision lets the generator pick a plausible-but-wrong pattern.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- For response-level hallucination detection in retrieval-augmented settings, having the model first convert the reference documents into a relational SQL database and then check the response against SELECT query results raises precision and macro F1 over zero-shot direct prediction, over atomic-claim decomposition, and over flat JSON key-value extraction, without any task-specific fine-tuning.Stating false facts confidently · reviewed by AI
- When post-training a 7-8B open model for text-to-SQL with a fixed budget of synthetic examples drawn from a large external pool, selecting examples whose schema-independent SQL skeletons are farthest from those already in the seed set (greedy K-center) rather than sampling randomly from the same pool yields a small in-domain gain and a larger cross-domain one (BIRD dev 64.9 vs 64.4, Spider Test 85.3 vs 83.4, Science Benchmark 53.9 vs 49.8).Writing a reusable locator: regex, SQL, XPath · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: high. Falsifier as drafted: A model whose database-split exact-match accuracy is close to its example-split accuracy on this corpus, or no decline in accuracy as foreign-key count grows. Drafted stance toward static-code-benchmarks-saturate-need-continuous-refresh: supports -- Both show that reusing the same programs or problems across train and test inflates measured synthesis ability.