within-one-model-family-relative-distribution-reasoning-failure-modes CLM-0511Within one model family, the relative distribution of reasoning failure modes stays similar from 1.5B to 72B, so critiques mined offline from small models' wrong answers, retrieved by failure mode and inserted as in-context exemplars, raise a large model's math accuracy above random, fixed, or embedding-similarity exemplar selection at a single generation.
Capability: Judging another's work as well as a careful reviewer would
Observed on
Qwen2.5 (32B, 72B) and Llama-3.1-70B targets with CritBank built from 1-8B same-family models; math reasoning (GSM8K, MATH, AMC23, AIME24/25), greedy decoding, 5-shot; some extensi.
Sources
- supportsCritICL: Inference-Time Weak-to-Strong Generalization from Small Language Model Failure ModesValidated by: Pass@1 on GSM8K, MATH, AMC23, AIME24/25 against zero/1/3/5-shot ICL, self-consistency@3/5/7, self-reflection and LLM-as-judge; CritICL-static reached 49.8 overall on Qwen2.5-32B vs 46.0 for 5-shot fixed and 49.5 for Consistency@7, at 1 generation and 3768 total tokens vs 5440; ablation swapping selection for random/fixed/semantic lost 4-6 points on AMC23/AIME25.. Measured on two Qwen targets and one Llama target over five math benchmarks with ICL and test-time-scaling baselines and a selection ablation; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For a frozen LLM run in a generate-critique-revise loop with no external feedback, the accuracy gain is concentrated where single-shot reasoning is already unreliable: on clean multi-step reasoning (BBH, ~74% single-shot) depth makes no difference beyond the confidence interval, while on harder math (MATH, 26.2% single-shot) the same loop gains +14.2 points, with GSM8K in between (+4.2).Fixing its own mistakes · reviewed by AI
- On statutory tax calculation cases, models asked to solve an input that contains an injected contradictory fact mostly compute straight through it and return the clean-input answer, while the same model asked only to verify the same input flags most of those contradictions \u2014 so abstention failure here is a task-framing gap, not missing detection ability.Saying it does not know · reviewed by AI
- General-purpose factuality verifiers (entailment-, consistency-, and RAG-trained) and zero-shot frontier LLM judges perform near chance at spotting unsupported claims in scientific peer reviews that must be checked against the full submitted paper, because they cannot separate paper-unsupported assertions from legitimate evaluative critique; small models fine-tuned on in-domain examples beat them by a wide margin.Stating false facts confidently · reviewed by AI
- For small open-weight Qwen2.5 models (1.5B-3B) doing multi-step math reasoning, features derived from the spread of attention entropy across layers at each generation step predict whether the final answer is wrong better than output-logit confidence features, with the margin larger on harder problems (+0.0093 AUC on GSM8K, +0.0759 on MATH-500 at 3B).Stating false facts confidently · reviewed by AI
- General-purpose factuality verifiers (entailment-, consistency-, and RAG-trained) and zero-shot frontier LLM judges perform near chance at spotting unsupported claims in scientific peer reviews that must be checked against the full submitted paper, because they cannot separate paper-unsupported assertions from legitimate evaluative critique; small models fine-tuned on in-domain examples beat them by a wide margin.Stating false facts confidently · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: Show that critique exemplars retrieved by failure mode do no better than semantically similar or random exemplars of the same length on the same benchmarks, or that small- and large-model failure mode distributions in a family are uncorrelated while the method still helps (or fails to help). Automatic check flagged: figures not in the source: 0.2.