chain-of-thought-faithfulness-measured-how-much-internal-concepts-sparse-autoencoder-features CLM-0368mechanismsingle paperReviewed by AI
When chain-of-thought faithfulness is measured by how much the internal concepts (sparse-autoencoder features) active during a CoT pass overlap with those active during a direct-answer pass, that overlap barely predicts whether the shared concepts causally drive the answer: across five open models and four reasoning datasets, above-median concept alignment raised the chance of an above-median causal ablation effect by only about 8%, so representational correspondence cannot substitute for an ablation test.
Capability: Saying what actually drove the answer
Observed on
Open models 1.7B-9B (Llama-3.1, Gemma-2, Qwen3) with off-the-shelf residual-stream SAEs, on GSM8K, LogiQA, OpenbookQA and ARC-Easy, measured at the answer-token position and only o.
Sources
- supportsFrom Concept Alignment to Causal Grounding: An Intervention Test of Chain-of-Thought FaithfulnessValidated by: Compared against random-subset, prediction-only, and norm-matched ablation controls (all near zero effect), a completeness check adding prediction-only features (negligible increment), a top-k magnitude upper bound (only marginally higher), and two SAE width/sparsity configurations (Pearson r > 0.8 across configurations).. Measured on five models and four datasets with layer-wise reporting; association quantified by median-binarised lift of 1.08 and weak Spearman correlations. Controls rule out ablation count, feature magnitude and SAE hyperparameters. [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- An abstention rule built from self-consistency, where the model samples several answers and itself scores how many agree with a candidate, can be thresholded by conformal prediction to bound the hallucination rate at a chosen level on closed-book QA; against log-probability confidence it abstains much less often on long-form answers (Temporal Sequences) for the same error bound and matches it on short answers (TriviaQA).Saying it does not know · reviewed by AI
- Prompting a model to behave as if its knowledge ended at an earlier date works for facts asked about directly (about 82% unlearning success) and for word-sense shifts (about 70%), but fails when the post-cutoff knowledge is only causally connected to the question (about 19%), so a simulated cutoff cannot be trusted to remove contamination from temporal prediction tests.Recalling facts it was trained on · reviewed by AI
- Instruction tuning that forces an answer to every question teaches models to fabricate when the question lies outside their parametric knowledge; splitting the tuning data by whether the base model already answers correctly and appending "I am unsure" to the rest (refusal-aware tuning) raises average precision on known-versus-unknown separation over vanilla tuning, transfers to out-of-domain datasets, and yields better calibrated confidence than post-hoc uncertainty scoring.Saying it does not know · reviewed by AI
- In 7-8B open-weight instruct models (Llama-3.1, Mistral-v0.3, Qwen2.5), the drop in a token's probability when a contiguous mid-layer MLP span is zero-ablated is near zero for function words, elevated for knowledge-bearing tokens, and abnormally spiked for hallucinated tokens, so this internal attribution signal — approximable by a small probe from one forward pass — can be used in beam scoring to raise factuality on open-ended and long-form generation, while leaving strict short-answer EM/F1 unchanged or worse.Stating false facts confidently · reviewed by AI
- For large audio language models on multiple-choice audio reasoning benchmarks (MMAR, MMAU-mini), accuracy does not track whether the model's chain of thought is acoustically grounded or causally drives the answer: several models answer correctly from perception steps that assert unsupported labels, and for some models replacing the reasoning chain with vacuous or fabricated content leaves the final answer unchanged.Understanding speech and audio · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Show that a concept-overlap score between CoT and prediction passes predicts ablation-based answer-probability drop with strong correlation across models, datasets and layers, so the causal ablation adds no information.