spoken-language-models-answering-questions-over-long-dialogue-histories CLM-0491observationsingle paperReviewed by AI
For spoken language models answering questions over long dialogue histories, converting the history to transcripts beats keeping it as audio when the answer is a lexical detail (exact slot values across dense turns: mean 71.3% vs 42.6%), but loses when the answer depends on how something was said, where audio history wins for six of seven models and by a larger margin once lexical emotion cues are removed.
Capability: Understanding speech and audio
Observed on
Seven spoken language models (Qwen3-Omni, GLM-4-Voice, AudioFlamingo3/Next, MiMo-Audio, Gemini-3.1-Pro, Qwen3.5-Plus) at a 32,768-token context, with text histories from SenseVoice.
Sources
- Validated by: Each model run twice on the same 13,516 QA pairs with text vs audio history; text won on Ultra Multi-Turn (71.3% vs 42.6% mean) and Personal Monologues, audio won on Beyond-Semantic for six of seven models, with the audio advantage rising from -0.6 to +6.6 points model-averaged when semantic emotion cues were controlled by synthesizing neutral text in emotional styles.. Measured on a purpose-built benchmark (1,474 sessions, 13,516 QA, 127.96h) with answer-provenance annotations; both modalities run on the same items, so the comparison is controlled. [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On multi-session dialogue questions whose answers depend only on how an utterance was delivered, a cascaded ASR-then-LLM pipeline (Whisper large-v3 into a frontier model) scores below 7B audio-native models on the same clips, and below even a transcript-only baseline, because the transcription step drops delivery cues entirely and adds content errors.Understanding speech and audio · reviewed by AI
- In audio language models that feed encoder outputs into a text LLM, downstream understanding accuracy across speech, sound and music is set by how semantically text-aligned the encoder is rather than by whether its representation is discrete or continuous — reconstruction-oriented codecs with high acoustic fidelity underperform, semantically distilled tokenizers can beat their own continuous teacher, and scaling the LLM backbone does not recover the lost information.Understanding speech and audio · reviewed by AI
- For large audio language models on multiple-choice audio reasoning benchmarks (MMAR, MMAU-mini), accuracy does not track whether the model's chain of thought is acoustically grounded or causally drives the answer: several models answer correctly from perception steps that assert unsupported labels, and for some models replacing the reasoning chain with vacuous or fabricated content leaves the final answer unchanged.Understanding speech and audio · reviewed by AI
- When large audio language models are asked to return every time span matching a natural-language query, they almost always report fewer intervals than exist: across 21 systems on queries averaging about 4 ground-truth intervals, under-reporting rates run from 79% to 100% and no system exceeds 13.2% exact count accuracy, even for the models with the best overlap scores.Understanding speech and audio · reviewed by AI
- On long-video (10-150 minute) audio-visual reasoning questions, multiple-choice accuracy overstates grounding: a large open model can answer about 55% of items correctly while scoring 8.5 on spatio-temporal evidence localization, whereas a frontier proprietary model at 73% accuracy scores 35.2 on the same evidence metric, so answer accuracy alone cannot distinguish deduction from prior-driven guessing.Reasoning about time in video · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A model set where audio history matches or beats transcripts on exact lexical retrieval from long histories, or where transcripts match audio on emotion questions whose text is semantically neutral.