ensembles-heterogeneous-vlms-answering-multimodal-multiple-choice-questions-entropy-uniformly CLM-0383mechanismsingle paperReviewed by AI
For ensembles of heterogeneous VLMs answering multimodal multiple-choice questions, the entropy of a uniformly pooled distribution over semantic-equivalence classes of sampled responses ranks wrong answers above right ones better than majority-vote entropy or picking the single lowest-entropy model, and the advantage grows with ensemble size; the same signal measured at the subagent evidence layer of a multi-step multi-agent system only ranks system failures slightly above chance.
Capability: Saying it does not know
Observed on
Training-free, no logits needed; 400 ScienceQA and 200 MMMU-Pro samples; 10 samples per model at temperature 1.0; three-model ensembles drawn from 8 small open, 6 large open, 6 com.
Sources
- Validated by: AUROC for prediction-error detection and AUARC for abstention against majority voting, naive lowest-entropy selection, and mean single-VLM reference: collective uncertainty 0.764 AUROC / 0.889 AUARC in the small regime (4.7-15.8 points over baselines), 0.906/0.987 large, JSD best in commercial at 0.819/0.910; pooled prediction accuracy up 13.0, 5.6, 8.7 points over the average single VLM; trajectory-level 0.619 AUROC / 0.699 AUARC.. [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On month-level egocentric recordings (20-120 days per participant) with human-written multiple-choice questions, current multimodal models degrade as the question moves from inferring recurring habits, to retrieving one specific episode, to composing evidence across days — the best model reached 71.8% macro-average against a 94.2% human baseline, with several models near or below the 25% chance level on route reasoning, cross-view spatial reasoning and direction judgement.Reasoning about time in video · reviewed by AI
- On multimodal cultural-interpretation multiple-choice questions, pipelines that pool several same-backbone evidence streams (perception, retrieval, web verification) and weight them by scored relevance beat majority voting, self-consistency and agent debate mainly on the items where the streams disagree, and the margin shrinks sharply as the backbone gets stronger.Getting more from several models than from one · reviewed by AI
- For binary visual question answering with open LVLMs (InstructBLIP-7B, Qwen2.5-VL, InternVL-4B), a hallucinated answer can be held with high response consistency, so a single consistency/uncertainty score misses it; separating consistency into four probe groups (image vs. text perturbation crossed with affirmative vs. negated statement) and classifying on the joint within- and between-group agreement pattern detects hallucination better than any single-score black-box or white-box baseline.Stating false facts confidently · reviewed by AI
- When a document image states multi-step selection constraints in narrative text and the supporting numbers sit in several charts, multimodal models degrade steadily as either the number of logical hops (2 to 5) or the number of charts (2 to 6) grows, with the best model reaching 62.83% overall against over 90% for human annotators.Carrying a chain of inference through to its conclusion · reviewed by AI
- On 2,154 expert-curated multiple-choice questions across eight architectural categories, 11 current models all score above 90% zero-shot (best about 94%), with quality-attribute and pattern questions near ceiling, but accuracy varies by category and drops for smaller models on the harder tiers (to about 85-87%), so declarative architectural knowledge is largely present while the benchmark cannot show whether it is applied in design.Making design decisions that hold up as a system grows · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An evaluation where pooled semantic entropy over several VLMs ranks errors no better than majority-vote entropy or the best single model's semantic entropy, or where its AUROC advantage shrinks rather than grows as models are added.