vision-language-models-given-explicit-abstention-option-plus-stated-payoff CLM-0654observationsingle paperReviewed by AI
When vision-language models are given an explicit abstention option plus a stated payoff for correct, wrong and declined answers, quadrupling the penalty for a wrong answer barely changes how often they answer, so their abstention rate is largely insensitive to the stated cost of error across text, synthetic video and natural video recall tasks.
Capability: Saying it does not know
Observed on
Nine pretrained VLM families on an associative-recall memory task (candidate occurred in stream or not) with two prompt arms differing only by an appended payoff sentence (+3/-1 vs.
Sources
- Measured: coverage R per model under two payoff arms on three evaluation subsets (synthetic text, synthetic video, natural video). Paper reports no model abstains meaningfully more under the costlier arm, and some abstain more when guessing is cheaper. Payoff-scored comparison shows small movement in the right direction and most dropped questions were ones answered wrong in the other arm, so selectivity exists while responsiveness does not. Single paper; abstention elicited only by prompt framing, not by any trained mechanism.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When Video-LLMs are asked to verify sentence by sentence whether a dense video caption is supported by the video, they detect hallucinated sentences less often in captions they generated themselves than in captions from other models, and detection of temporal errors (action, order, causality, frequency, state change) improves much less with model scale than detection of static-frame errors.Reasoning about time in video · reviewed by AI
- On long-video (10-150 minute) audio-visual reasoning questions, multiple-choice accuracy overstates grounding: a large open model can answer about 55% of items correctly while scoring 8.5 on spatio-temporal evidence localization, whereas a frontier proprietary model at 73% accuracy scores 35.2 on the same evidence metric, so answer accuracy alone cannot distinguish deduction from prior-driven guessing.Reasoning about time in video · reviewed by AI
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
- An abstention rule built from self-consistency, where the model samples several answers and itself scores how many agree with a candidate, can be thresholded by conformal prediction to bound the hallucination rate at a chosen level on closed-book QA; against log-probability confidence it abstains much less often on long-form answers (Temporal Sequences) for the same error bound and matches it on short answers (TriviaQA).Saying it does not know · reviewed by AI
- Large pretrained models are well calibrated on multiple-choice and true/false questions when answer options are shown as lettered choices, and can predict the probability that their own sampled answer is correct (P(True)) or that they know the answer at all (P(IK)); calibration improves with model size and few-shot prompting, self-evaluation improves when the model sees several of its own samples first, and P(IK) generalizes only partly to new tasks, where its calibration degrades.Saying it does not know · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Showing that models materially lower coverage when the wrong-answer penalty rises, under the same two-arm prompt protocol, on these or comparable recall tasks.