multimodal-model-given-external-text-contradicts-image-making-commit CLM-0105When a multimodal model is given external text that contradicts the image, making it commit to a description of the image before the text is admitted, then reconciling the two in a second call, recovers accuracy the single-prompt setup loses — but the benefit of withholding the text from the first call is model- and text-source dependent, and on some models context-preserving prompting is still better.
Capability: Telling the user what they want to hear
Observed on
Image-question pairs where the visible scene conflicts with commonsense priors and a single false sentence supports the prior; six frontier multimodal models; false text generated .
Sources
- 998-case constructed diagnostic, 499 abnormal images with false text; six models. Arbitration over the isolated witness gains 19.7-44.1 points with paired 95% CIs excluding zero. But Kimi-K2.5 does best under a plain visual-supremacy prompt and Qwen3-Instruct under joint conditioning; on the GPT-4o-regenerated 100-case subset joint is best for three of four models. Ground truth is the image answer by construction, and grading is by a GPT-4o-mini judge with partial human audit. Adversarial by design, not a prevalence estimate.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When vision-language models are given an explicit abstention option plus a stated payoff for correct, wrong and declined answers, quadrupling the penalty for a wrong answer barely changes how often they answer, so their abstention rate is largely insensitive to the stated cost of error across text, synthetic video and natural video recall tasks.Saying it does not know · reviewed by AI
- When a document image states multi-step selection constraints in narrative text and the supporting numbers sit in several charts, multimodal models degrade steadily as either the number of logical hops (2 to 5) or the number of charts (2 to 6) grows, with the best model reaching 62.83% overall against over 90% for human annotators.Carrying a chain of inference through to its conclusion · reviewed by AI
- When an 8B model verifies natural-language claims against knowledge-graph triples, scoring the evidence twice under opposing prompts (argue it supports, argue it refutes) and adjudicating between the two arguments avoids the bias toward a "Supported" verdict that a single-perspective assessment shows, and is the largest single contributor to accuracy in that pipeline.Checking claims against evidence · reviewed by AI
- When a prompt contains several legitimate but conflicting evidence passages, models internally register the conflict — linear probes on mid-to-late-layer hidden states separate conflicting from consistent inputs at high AUC — while still resolving the conflict badly at the output, so the failure is in evidence integration rather than in conflict detection.Checking claims against evidence · reviewed by AI
- When training an 8B multimodal model with RL to find scientific errors in whole papers without being told where to look, mixing in matched easier variants of the same instances — where the error is asserted and the evidence supplied — yields better performance on the evidence-absent task itself than training only on that task at matched exposure.Spotting an error in a given artifact · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A replication where the staged witness-then-arbiter pipeline gives no gain over a prompt-matched single-call condition on conflicting image-text cases, or where withholding the text from the witness helps uniformly across models and text generators. Drafted stance toward withholding-the-users-opinion-is-the-no-training-alternative: supports -- Extends the same lever — keep the biasing text out of context when the answer is formed — to conflicting external text in multimodal input, while showing the withholding step alone is not what carries the gain. Proposed technique, not catalogued: Context-blind visual witness then arbitration.