arxiv-1809-08887 SRC-0250 · paper

Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task

Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li

Created: 2018-09-24 · Ingested: 2026-09-19

We present Spider, a large-scale, complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 college students. It consists of 10,181 questions and 5,693 unique complex SQL queries on 200 databases with multiple tables, covering 138 different domains. We define a new complex and cross-domain semantic parsing and text-to-SQL task where different complex SQL queries and databases appear in train and test sets. In this way, the task requires the model to generalize well to both new SQL queries and new database schemas. Spider is distinct from most of the previous semantic parsing tasks because they all use a single database and the exact same programs in the train set and the test set. We experiment with various state-of-the-art models and the best model achieves only 12.4% exact matching accuracy on a database split setting. This shows that Spider presents a strong ch…

In brief

Text-to-SQL accuracy on prior benchmarks was largely memorization: when target queries and databases are held out, the same models collapse. Spider makes that explicit.

The dataset has 10,181 questions and 5,693 unique complex SQL queries over 200 databases spanning 138 domains, written and cross-reviewed by 11 computer science students over about 1,000 hours. Each database averages 27.6 columns and 8.8 foreign keys, and the corpus contains 1335 ORDER BY, 1491 GROUP BY, 844 nested and 388 HAVING queries, against 0 of any of those in WikiSQL. 5 systems were run: Seq2Seq, +Attention, +Copying, SQLNet and TypeSQL, under an example split (8659/1034/2147) and a database split (146/20/40 databases).

On the example split TypeSQL reaches 33.0 exact match; on the database split the best is SQLNet at 12.4, and TypeSQL drops to 8.2. WHERE clauses are hardest (19.8 F1 for SQLNet under database split), and accuracy falls as the number of foreign keys rises.

The split comparison isolates generalization to unseen schemas cleanly, since the same models and data generation are held fixed. Models predate large pretrained LMs, values are not predicted, ambiguous and commonsense questions were excluded, and execution accuracy is not reported.

Treat pre-Spider text-to-SQL numbers as inflated.

Written from the abstract by claude-opus-5 on 2026-10-17, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsOn natural-language-to-SQL synthesis over multi-table databases, a model's accuracy overstates its ability whenever the same database or the same query appears in both training and test: splitting by database rather than by example cut the best system's exact-match accuracy from 33.0% to 12.4%, and accuracy falls further as the target database has more foreign keys.