prompted-text-to-sql-agent-blanket-output-shaping-instruction-system-prompt-here CLM-0472mechanismsingle paperReviewed by AI
In a prompted text-to-SQL agent, a blanket output-shaping instruction in the system prompt (here "LIMIT 10") is applied by the model to every generated query regardless of the question's intent, truncating answers that legitimately need many rows, so removing it raises execution accuracy most on questions whose correct result set is large.
Capability: Writing a reusable locator: regex, SQL, XPath
Observed on
Single open-weight model (gpt-oss-120b) at temperature 0, prompt engineering only, one hand-built 100-question benchmark over one PostgreSQL database (Chicago Crime); no external b.
Sources
- supportsAn Iterative LangGraph Agent for Text-to-SQL: Natural Language Access to the Chicago Crime DatabaseValidated by: Two prompt revisions of the same agent on 100 stratified questions: overall execution accuracy 47%->60% under a hybrid relational equivalence metric (12%->19% strict JSON match), valid SQL rate 87%->93%, synthesis quality 3.91->4.34; hard-tier strict EX 7%->17%. Confounded with a concurrent relevance-checker keyword expansion.. Measured but weakly isolated: V1 vs V2 changed both the LIMIT instruction and the relevance-checker keyword list, so the two effects are not fully separated; [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On text-to-SQL questions that are underspecified with respect to the schema, retrieving clause-level fragments (conditions, joins, calculations, groupings, output specs) from a database's own past successful SQL queries and prepending them to existing few-shot pipelines raises execution accuracy across four different pipelines, but the gain saturates and can reverse once too many candidates are retrieved, because high recall with low precision lets the generator pick a plausible-but-wrong pattern.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- For instruction tuning an 8B open model on text-to-SQL, a curated set of under 900 examples selected to cover every table in the target schemas and score well on question-SQL alignment reaches higher execution accuracy than pipelines trained on tens of thousands to millions of samples, and dropping the schema-coverage term from the selection objective costs more accuracy than dropping any single quality term.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
- Instruction tuning that forces an answer to every question teaches models to fabricate when the question lies outside their parametric knowledge; splitting the tuning data by whether the base model already answers correctly and appending "I am unsure" to the rest (refusal-aware tuning) raises average precision on known-versus-unknown separation over vanilla tuning, transfers to out-of-domain datasets, and yields better calibrated confidence than post-hoc uncertainty scoring.Saying it does not know · reviewed by AI
- On factual QA (817-item TruthfulQA multiple-choice, plus a 300-item Natural Questions short-answer set), a prompt-only three-stage gate in which the model lists the information units a question needs, scores its support for each, and answers only when the mean score clears a threshold lowers the unconditional wrong-answer rate and raises accuracy on answered items for all eleven models tested, but most of the abstentions it produces fall on questions plain chain-of-thought answered correctly (about 69% at the primary operating point), so the gain is bought with lost coverage rather than with selective self-knowledge.Saying it does not know · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Removing a blanket row-limit instruction leaves execution accuracy unchanged, or the gain is concentrated on single-row-answer questions rather than multi-row ones, in a controlled single-variable comparison.