Understanding Syllogistic Reasoning in LLMs from Formal and Natural Language Perspectives

The strongest models judge syllogisms by logical form and ignore whether the conclusion is believable, the opposite of the human belief-bias pattern. Weaker models show the human-like bias.

The setup is 160 syllogisms built from 40 handcrafted base items plus 3 variants each (nonsense predicates, premise order switched, both), each labelled twice: syntactic validity and real-world believability. 14 models from 8 organizations (1B to 671B, dense and MoE) were run across 4 prompting strategies (zero-shot, one-shot, few-shot, zero-shot CoT) and 3 temperatures, for 26,880 evaluations.

Syntax accuracy averaged 81.7% against 56.2% NLU accuracy, a 25.50pp gap. Gemini 2.5 Flash reached 99.6% syntax and 51.7% NLU. 12 of 14 models scored higher on congruent than incongruent items, mean +10.81 pp (p = 0.0280, d = 0.66), ranging from +0.8 to +46.9. Bias shrank with accuracy (ρ = −0.565). Few-shot underperformed zero-shot by 3.57 pp (p = 0.0165), surviving Holm-Bonferroni, though the Friedman test across all 4 strategies showed no effect (p = 0.356).

Effects are measured with within-model comparisons and instance-level McNemar tests, so the bias contrast is well isolated; the prompting result is fragile across tests. Scope is categorical syllogisms only, 14 models, no nested quantifiers, modal operators, or adversarial distractors, and no mechanistic account of the near-perfect syntax performance.

Treat belief-bias findings from small Llama-class models as unlikely to transfer to frontier models.