multiple-choice-audio-questions-where-each-answer-choice-paired-generated CLM-0106On multiple-choice audio questions where each answer choice is paired with a generated image, adding the images helps native multimodal models most on environmental-sound questions and not at all on speech (speech is best audio-only), and in cascaded pipelines that convert audio to transcripts or captions first the images lower average accuracy.
Capability: Understanding speech and audio
Observed on
5,667-item multiple-choice benchmark over speech, sound, music and mixed audio in six languages, with Stable-Diffusion/GPT-image-generated images for each choice; models tested inc.
Sources
- Accuracy table across 6 languages and 3 audio domains, plus a modality-ablation of the fine-tuned model. Qwen-2.5-omni gains +15.55% from audio-only to audio+image (p<0.01 paired t-test) with sound gaining most (+11.7%); cascaded Gemma3n drops when images are added (59.91 to 59.00 average); ablation shows speech best audio-only in most languages. Images are synthetic and one per choice, so the setup may leak answer information and does not isolate scene grounding from choice-level visual cues.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On multi-session dialogue questions whose answers depend only on how an utterance was delivered, a cascaded ASR-then-LLM pipeline (Whisper large-v3 into a frontier model) scores below 7B audio-native models on the same clips, and below even a transcript-only baseline, because the transcription step drops delivery cues entirely and adds content errors.Understanding speech and audio · reviewed by AI
- For large audio language models on multiple-choice audio reasoning benchmarks (MMAR, MMAU-mini), accuracy does not track whether the model's chain of thought is acoustically grounded or causally drives the answer: several models answer correctly from perception steps that assert unsupported labels, and for some models replacing the reasoning chain with vacuous or fabricated content leaves the final answer unchanged.Understanding speech and audio · reviewed by AI
- For spoken language models answering questions over long dialogue histories, converting the history to transcripts beats keeping it as audio when the answer is a lexical detail (exact slot values across dense turns: mean 71.3% vs 42.6%), but loses when the answer depends on how something was said, where audio history wins for six of seven models and by a larger margin once lexical emotion cues are removed.Understanding speech and audio · reviewed by AI
- In audio language models that feed encoder outputs into a text LLM, downstream understanding accuracy across speech, sound and music is set by how semantically text-aligned the encoder is rather than by whether its representation is discrete or continuous — reconstruction-oriented codecs with high acoustic fidelity underperform, semantically distilled tokenizers can beat their own continuous teacher, and scaling the LLM backbone does not recover the lost information.Understanding speech and audio · reviewed by AI
- For 4-8B multimodal models on long-video QA, RL training on questions authored from short scene segments but answered over the whole video improves benchmark accuracy more than RL on questions authored from the whole video, because segment-authored questions target details a whole-video view overlooks; including the segment timestamp in the training instruction adds a further gain by removing referential ambiguity, and is not needed at inference.Reasoning about time in video · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: On the same benchmark, native multimodal models show no sound-domain gain from the paired images, or the gain is as large for speech as for sound, or cascaded transcript/caption pipelines also improve when images are added. Proposed technique, not catalogued: Image-per-answer-choice grounding for audio MCQ.