models-judge-syllogism-s-validity-partly-how-believable-conclusion CLM-0311mechanismsingle paperReviewed by AI
Models judge a syllogism's validity partly by how believable its conclusion is because validity and plausibility are encoded as linearly separable but strongly aligned directions in the residual stream: steering along the plausibility direction causally shifts validity judgements and vice versa, the degree of alignment predicts the size of the content effect across models, and a debiasing vector that separates the two reduces the effect and raises accuracy.
Capability: Deducing validly from logical structure
Observed on
Qwen-2.5 and Qwen3 families among others, syllogism datasets with believable and unbelievable conclusions, 2025; steering and probing on open weights..
Sources
- Validated by: Content-effect size and accuracy before and after steering across model sizes; causal steering in both directions; alignment-effect correlation across models. Open models only.. A mechanistic account with causal steering, which is stronger than the usual behavioural report of content effects; whether the debiasing vector generalizes beyond syllogisms is untested.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Presenting syllogisms as Euler or linear diagrams instead of text or logical notation did not consistently improve two models' accuracy on 285 problems; both did well on entailment and contradiction items but poorly on neutral ones (GPT-4o-mini under 30% on neutral cases), and made systematic conversion errors, treating 'all A are B' as if it implied 'all B are A'.Deducing validly from logical structure · reviewed by AI
- General-purpose factuality verifiers (entailment-, consistency-, and RAG-trained) and zero-shot frontier LLM judges perform near chance at spotting unsupported claims in scientific peer reviews that must be checked against the full submitted paper, because they cannot separate paper-unsupported assertions from legitimate evaluative critique; small models fine-tuned on in-domain examples beat them by a wide margin.Stating false facts confidently · reviewed by AI
- Across 14 models, syllogisms posed in symbolic form are solved far better than the same inferences in natural language: top models reach near-perfect symbolic accuracy (about 99.6%) while scoring around chance (52%) on the natural-language understanding variant, mean 81.7% versus 56% overall, and few-shot prompting did not help, so formal-pattern competence does not carry over to reading the logic out of ordinary sentences.Deducing validly from logical structure · reviewed by AI
- When an 8B model verifies natural-language claims against knowledge-graph triples, scoring the evidence twice under opposing prompts (argue it supports, argue it refutes) and adjudicating between the two arguments avoids the bias toward a "Supported" verdict that a single-perspective assessment shows, and is the largest single contributor to accuracy in that pipeline.Checking claims against evidence · reviewed by AI
- An abstention rule built from self-consistency, where the model samples several answers and itself scores how many agree with a candidate, can be thresholded by conformal prediction to bound the hallucination rate at a chosen level on closed-book QA; against log-probability confidence it abstains much less often on long-form answers (Temporal Sequences) for the same error bound and matches it on short answers (TriviaQA).Saying it does not know · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Models where the two directions are already orthogonal yet content effects persist, which would mean the mechanism is elsewhere.