statutory-tax-calculation-cases-models-asked-solve-input-contains CLM-0642On statutory tax calculation cases, models asked to solve an input that contains an injected contradictory fact mostly compute straight through it and return the clean-input answer, while the same model asked only to verify the same input flags most of those contradictions \u2014 so abstention failure here is a task-framing gap, not missing detection ability.
Capability: Saying it does not know
Observed on
Six recent LLMs on 91 SARA-derived US federal income tax cases; contradictions injected into categorical facts (marital/filing status, dependent count); missing-fact perturbations .
Sources
- Validated by: Across all six models, contradiction abstention rose from 0-43% (solver alone) to 70-83% with the gate, at a clean-accuracy cost of 0-5.1 percentage points; false-flag rate on clean inputs 0-5.5%. Compared against a self-consistency baseline over three solve runs, which flagged only 11-35% of contradictions at a higher false-flag rate.. Measured on 108 contradiction variants and 78 redacted variants from 91 SARA cases, three runs per model, binary numeric-or-null output schema. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Within one model family, the relative distribution of reasoning failure modes stays similar from 1.5B to 72B, so critiques mined offline from small models' wrong answers, retrieved by failure mode and inserted as in-context exemplars, raise a large model's math accuracy above random, fixed, or embedding-similarity exemplar selection at a single generation.Judging another's work as well as a careful reviewer would · reviewed by AI
- When a tool call is forced and returns an unusable payload, tool-augmented agents report the failure honestly almost always if the response envelope declares an error, but assert unsupported values or invent policy reasons in a large share of cases when the envelope says status:ok while withholding the value (redacted, corrupted, stale, empty, malformed, truncated).Stating false facts confidently · reviewed by AI
- When an error is injected into a supplied chain of thought, models mostly carry it to the answer: a planted arithmetic error costs small models 50-60% accuracy and 18% on average, though the largest models lose only 3-7%; unit-conversion errors cost about 9% at every scale; skipped steps and sycophantic steps cost small models about 10%; extra correct steps cost almost nothing (0-6%).Carrying a chain of inference through to its conclusion · reviewed by AI
- When vision-language models are given an explicit abstention option plus a stated payoff for correct, wrong and declined answers, quadrupling the penalty for a wrong answer barely changes how often they answer, so their abstention rate is largely insensitive to the stated cost of error across text, synthetic video and natural video recall tasks.Saying it does not know · reviewed by AI
- An abstention rule built from self-consistency, where the model samples several answers and itself scores how many agree with a candidate, can be thresholded by conformal prediction to bound the hallucination rate at a chosen level on closed-book QA; against log-probability confidence it abstains much less often on long-form answers (Temporal Sequences) for the same error bound and matches it on short answers (TriviaQA).Saying it does not know · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A setup where models asked to verify the same contradicted inputs flag them no more often than they abstain while solving, or where the verification gate fails to raise contradiction abstention without large clean-accuracy loss. Automatic check flagged: figures not in the source: 014.