long-chain-thought-verified-classifying-graph-steps-predictive-signal CLM-0428mechanismsingle paperReviewed by AI
When a long chain of thought is verified by classifying a graph of its steps, the predictive signal comes almost entirely from the semantic content of the steps, not the graph shape: replacing step embeddings with purely structural metadata (child/parent counts, chain index, position) drops accuracy to near chance on math, science and multiple-choice reasoning, while remaining above chance only on code.
Capability: Carrying a chain of inference through to its conclusion
Observed on
8,000 long chains of thought from three open reasoning models (DeepSeek-R1 distills, QwQ-32B) over MATH, GPQA, LiveCodeBench-v5 and MMLU-Pro; correctness of the final answer is the.
Sources
- supportsLCoT-GV: Graph Attention Networks for Verifying Long Reasoning Chains in Large Language ModelsValidated by: Accuracy averaged over five runs against LCoT2Tree and a length-based classifier on four benchmarks: average 77.03 for the proposed method vs 74.68 for LCoT2Tree, with gains on LiveCodeBench and GPQA and losses on MATH and MMLU-Pro.. Measured ablation: metadata-only features scored roughly 53 average vs 75-78 for embedding features across three generating models, with LiveCodeBench the exception (66-70). Single paper, one graph-construction pipeline, chain-level rather than step-level labels; [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When a document image states multi-step selection constraints in narrative text and the supporting numbers sit in several charts, multimodal models degrade steadily as either the number of logical hops (2 to 5) or the number of charts (2 to 6) grows, with the best model reaching 62.83% overall against over 90% for human annotators.Carrying a chain of inference through to its conclusion · reviewed by AI
- For 0.6B-8B Qwen3 models on math, logic and multiple-choice reasoning, quantifying reasoning uncertainty from a directed acyclic graph built by sampling several chains and merging semantically equivalent steps with an NLI model separates correct from incorrect generations better than sequence-level entropy, likelihood and self-prompting uncertainty measures.Carrying a chain of inference through to its conclusion · reviewed by AI
- Verifying a chain of thought by decomposing it into nodes and asking binary consistency questions per node localizes errors better than judging the whole chain at once (F1 up 10-25% over CoT-based verifiers, error-localization accuracy far above an 8-vote end-to-end judge) while using 6-58 times fewer tokens, because whole-chain judging dilutes attention and multi-sampling pays for long generations.Carrying a chain of inference through to its conclusion · reviewed by AI
- For LLM agents classifying nodes in text-attributed graphs, serializing sampled neighborhoods into prompt text makes the prediction depend on the arbitrary order of neighbors; replacing that text with a fixed-size, permutation-invariant vector of graph statistics (degree, PageRank, clustering, cross-community fraction, label entropy, majority-label fraction) plus only the top-k semantically relevant node texts raises accuracy and cuts per-message tokens.Getting more from several models than from one · reviewed by AI
- An abstention rule built from self-consistency, where the model samples several answers and itself scores how many agree with a candidate, can be thresholded by conformal prediction to bound the hallucination rate at a chosen level on closed-book QA; against log-probability confidence it abstains much less often on long-form answers (Temporal Sequences) for the same error bound and matches it on short answers (TriviaQA).Saying it does not know · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A structure-only feature set (no step text embeddings) reaching accuracy close to the embedding-based verifier on these same chain-correctness datasets.