explanation-faithfulness CAP-0015 · activeSaying what actually drove the answer
Whether a model's stated reasoning reflects the computation that produced its answer, rather than a plausible story told after the fact.
Tags: behavior, evaluation
A model can write a chain of thought that reads as sound while the answer was actually driven by something the explanation never mentions. This is not the same as being wrong: the answer can be right and the stated reasoning still not be the reason. It matters wherever an explanation is used as evidence — debugging an agent, auditing a decision, or filing a claim from a paper's reported reasoning.
What counts as this capability
Scope boundary used when deciding whether a paper is really about this capability, rather than merely mentioning it.
In scope when the paper tests whether stated reasoning corresponds to the process that produced the output: perturbing an unmentioned feature and watching the answer move, measuring whether steps are load-bearing, or auditing explanations against internal computation. NOT in scope: whether the reasoning is correct, which is the underlying task capability; and not interpretability work that reads internals without comparing them to what the model said.
Claims
- mechanismsingle paper↑1↓0When chain-of-thought faithfulness is measured by how much the internal concepts (sparse-autoencoder features) active during a CoT pass overlap with those active during a direct-answer pass, that overlap barely predicts whether the shared concepts causally drive the answer: across five open models and four reasoning datasets, above-median concept alignment raised the chance of an above-median causal ablation effect by only about 8%, so representational correspondence cannot substitute for an ablation test.
- observationsingle paper↑1↓0In event-conditioned time-series prediction (finance and traffic), models cite the factor that a controlled intervention actually manipulated in nearly all cases while their predictions change with that factor far less often — on traffic series-side pairs citation exceeds 90% for most of ten models but pair-level behavioral support stays under 22% — so citation of the right factor is not evidence the prediction depended on it.
- mechanismsingle paper↑1↓0When an LLM answers a question and gives an explanation, removing from the input the concepts its explanation did not credit and re-querying on the reduced input raises measured explanation faithfulness over both standard prompting and prompts that ask for faithfulness, because the removal deletes the unmentioned influences rather than asking the model to report them.
- mechanismsingle paper↑1↓0When an LLM gives an answer plus an explanation, deleting from the input those concepts the explanation does not credit and re-querying on the reduced input raises measured faithfulness over both standard prompting and prompts that ask for faithful explanations, because the re-query removes the unmentioned factors that could have influenced the original answer.
- observationsingle paper↑1↓0For open-source VLMs asked to justify answers on object-removal counterfactual image pairs (A-OKVQA, SNLI-VE), explanations written after the answer is committed (Predict-then-Explain) mention the removed object more in proportion to how much its removal shifts the answer distribution than pre-answer step-by-step CoT does, so pre-answer reasoning is not the more faithful format here.
- observationsingle paper↑1↓0On paired solvable/unsolvable math problems translated into French and Greek, natively multilingual models (Gemma-4-31B, Qwen3-4B/30B) encode solvability most sharply in English yet agree least with that internal belief in their stated verdict in English, being more faithful in the lower-resource languages, while a weakly multilingual model (Llama-3.1-8B) shows the opposite ordering.
- mechanismsingle paper↑1↓0In RL-trained multimodal image-quality models, applying the reasoning reward only to reasoning tokens and the human-rating reward only to rating tokens keeps rating alignment at the level of rating-only training while turning reasoning-guided edit quality gain positive, and it prevents the collapse to a single universal edit that an unmasked shared reward produces despite that variant's larger raw gain.
- mechanismsingle paper↑1↓0In RL-trained multimodal image-quality models, supervising reasoning with the rating reward alone raises rating alignment while making the stated reasoning less faithful — rating-only training reached an average PLCC/SRCC of 0.836/0.815 yet its reasoning-guided edits lowered judged image quality (mean gain -0.260, worse than the untrained baseline's -0.136).
Techniques
- Ablate CoT/prediction shared SAE concepts to test faithfulneprocessEncode the answer-position residual of a direct-prediction pass and a CoT pass with one shared sparse autoencoder, then zero the decoder contribution of the concepts active in both and measure the drop in answer probability.
- Re-query after removing concepts the explanation ignoredpromptingIdentify input concepts not credited in the model's explanation, delete them from the input, and re-ask the model so the final answer can only depend on the concepts it named.
- Reward reasoning by whether editing on it improves the imagetrainingAn actor states quality-limiting factors, a frozen diffusion editor edits the image conditioned on that reasoning, a frozen judge scores the quality change, and that change rewards only the reasoning tokens while human ratings reward only t
- Score cited factors by whether the paired intervention movesprocessFor each controlled pair that changes one cited factor, check both whether the explanation mentions that factor and whether both endpoint predictions follow the expected transition, reporting the conditional rate (CSDR).
Sources
Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.
- ↑ supports a claim here
- From Concept Alignment to Causal Grounding: An Intervention Test of Chain-of-Thought Faithfulness2026-09-19↑ supports a claim here
- about this capability · no claim drafted yet
- Counterfactual Tests for Measuring Chain-of-Thought Faithfulness in Visual Language Models2026-09-06↑ supports a claim here
- ↑ supports a claim here · ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here · ↑ supports a claim here
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.