prompt-contains-several-legitimate-conflicting-evidence-passages-models-internally CLM-0243When a prompt contains several legitimate but conflicting evidence passages, models internally register the conflict — linear probes on mid-to-late-layer hidden states separate conflicting from consistent inputs at high AUC — while still resolving the conflict badly at the output, so the failure is in evidence integration rather than in conflict detection.
Capability: Checking claims against evidence
Observed on
Measured on open-weight models where activations are accessible (Llama-3.1-8B-Instruct, GPT-OSS-20B); output-level failure measured across seven models from 8B to 120B on a multi-d.
Sources
- Validated by: Not stated. Layer-wise logistic-regression probes on paired conflict/consistent inputs give AUC above 0.85 in mid-to-late layers for most of six conflict types, replicated on a second architecture; same models score below 50% on inferential conflicts and show Gini imbalance in summarization. The dissociation is argued from two separate measurements, not from a single controlled manipulation, and probe separability may reflect surface cues rather than functional awareness.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · reviewed by AI
- When an 8B model verifies natural-language claims against knowledge-graph triples, scoring the evidence twice under opposing prompts (argue it supports, argue it refutes) and adjudicating between the two arguments avoids the bias toward a "Supported" verdict that a single-perspective assessment shows, and is the largest single contributor to accuracy in that pipeline.Checking claims against evidence · reviewed by AI
- On multimodal cultural-interpretation multiple-choice questions, pipelines that pool several same-backbone evidence streams (perception, retrieval, web verification) and weight them by scored relevance beat majority voting, self-consistency and agent debate mainly on the items where the streams disagree, and the margin shrinks sharply as the backbone gets stronger.Getting more from several models than from one · reviewed by AI
- In closed-book multi-hop QA the accuracy gain from decomposed prompting shrinks to little in frontier models, but answers still change between task-equivalent prompt regimes (agreement only about 70%, and about 60% on MuSiQue), and because stable facts survive re-prompting while hallucinations do not, disagreement across regimes detects errors better than standard uncertainty baselines on F1 and AUROC without retrieval or training.Recalling facts it was trained on · reviewed by AI
- When automatically rewriting the prompts of a multi-agent LLM system from its failures, locating the responsible agent by injecting a hint into one agent at a time and using that agent's hint-corrected intermediate output as the supervision signal produces prompt updates that generalize better than critique derived only from the system-level output.Improving its own scaffolding from failures · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Show that probes cannot separate conflicting from consistent inputs above chance in models that nonetheless resolve conflicts well, or that probe separability tracks conflict-resolution accuracy across models and conflict types. Drafted stance toward first-evidence-dominance-is-representational-not-decoding: supports -- The paper finds combined-evidence activations shift toward the first evidence at every layer, that random reordering does not remove the dominance, and that steering activations toward the neutral center improves accuracy.