0-6b-8b-qwen3-models-math-logic-multiple-choice-reasoning-quantifying CLM-0523For 0.6B-8B Qwen3 models on math, logic and multiple-choice reasoning, quantifying reasoning uncertainty from a directed acyclic graph built by sampling several chains and merging semantically equivalent steps with an NLI model separates correct from incorrect generations better than sequence-level entropy, likelihood and self-prompting uncertainty measures.
Capability: Carrying a chain of inference through to its conclusion
Observed on
Qwen3 0.6B-8B; GSM8K, MATH-500, AMC2022-2024, FOLIO, MMLU-Pro; ~9-10 sampled chains at temperature 1.0; selective-generation metrics (PRR, AUROC, AUPRC).
Sources
- Validated by: Selective-generation PRR/AUROC/AUPRC against 45 white- and black-box uncertainty baselines across 4 Qwen3 scales and 5 datasets, with 1000-fold bootstrap; reported average gains of 11.79% PRR, 13.33% AUROC, 9.66% AUPRC over the best contender; ablations show node merging carries most of the signal for the propagation variant.. Measured on 4 Qwen3 scales and 5 datasets against 45 baselines with bootstrap error bars; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In closed-book multi-hop QA the accuracy gain from decomposed prompting shrinks to little in frontier models, but answers still change between task-equivalent prompt regimes (agreement only about 70%, and about 60% on MuSiQue), and because stable facts survive re-prompting while hallucinations do not, disagreement across regimes detects errors better than standard uncertainty baselines on F1 and AUROC without retrieval or training.Recalling facts it was trained on · reviewed by AI
- When a long chain of thought is verified by classifying a graph of its steps, the predictive signal comes almost entirely from the semantic content of the steps, not the graph shape: replacing step embeddings with purely structural metadata (child/parent counts, chain index, position) drops accuracy to near chance on math, science and multiple-choice reasoning, while remaining above chance only on code.Carrying a chain of inference through to its conclusion · reviewed by AI
- When a document image states multi-step selection constraints in narrative text and the supporting numbers sit in several charts, multimodal models degrade steadily as either the number of logical hops (2 to 5) or the number of charts (2 to 6) grows, with the best model reaching 62.83% overall against over 90% for human annotators.Carrying a chain of inference through to its conclusion · reviewed by AI
- Verifying a chain of thought by decomposing it into nodes and asking binary consistency questions per node localizes errors better than judging the whole chain at once (F1 up 10-25% over CoT-based verifiers, error-localization accuracy far above an 8-vote end-to-end judge) while using 6-58 times fewer tokens, because whole-chain judging dilutes attention and multi-sampling pays for long generations.Carrying a chain of inference through to its conclusion · reviewed by AI
- For agentic tool calls where a parameter is fabricated rather than grounded in the dialogue, sampling-based uncertainty detectors (token entropy, semantic entropy, SEP) fail at near-chance because the model fabricates plausible parameters with high confidence, while the model's hidden states at the moment the parameter is generated do carry a separable grounding signal.Stating false facts confidently · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: On the same models and datasets, a sequence-level or self-prompted uncertainty baseline matching or beating graph-complexity scores on AUROC/PRR, or the graph advantage vanishing once node merging is replaced by a trivial grouping.