audio-temporal-localization-tasks-11-southeast-asian-languages-current-audio CLM-0363On audio temporal-localization tasks in 11 Southeast Asian languages, current audio LLMs return time spans whose coverage of the true span consistently exceeds their purity, and F1 falls steadily as the recording grows from under 30 seconds to 3 minutes, with several models unable to run at all on the longer bins.
Capability: Understanding speech and audio
Observed on
Audio/multimodal LLMs from 2B to 30B plus Gemini 2.5 Flash and GPT-4o, on content-to-time queries over recordings up to 3 minutes in SEA languages; English prompts..
Sources
- supportsSEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast AsiaMeasured on 13,145 samples per temporal task across five languages, with span-overlap F1 stratified into four duration bins; coverage-vs-purity decomposition shown in detail only for MERaLiON-2-10B and asserted as a pattern for most models. Single benchmark, no independent replication, and possible test contamination is acknowledged by the authors.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In audio in-context learning with per-episode resampled rules, current large audio language models bind arbitrary non-speech sounds to new labels reasonably well (best model 58.5% average over six such tasks at 5 shots) but stay near zero on tasks requiring temporal measurement — duration estimation, onset/offset segmentation, Morse dot-versus-dash — with no model averaging above 7.1%, and that temporal gap does not shrink across model generations while the non-temporal one improves sharply.Understanding speech and audio · reviewed by AI
- When large audio language models are asked to return every time span matching a natural-language query, they almost always report fewer intervals than exist: across 21 systems on queries averaging about 4 ground-truth intervals, under-reporting rates run from 79% to 100% and no system exceeds 13.2% exact count accuracy, even for the models with the best overlap scores.Understanding speech and audio · reviewed by AI
- On multi-session dialogue questions whose answers depend only on how an utterance was delivered, a cascaded ASR-then-LLM pipeline (Whisper large-v3 into a frontier model) scores below 7B audio-native models on the same clips, and below even a transcript-only baseline, because the transcription step drops delivery cues entirely and adds content errors.Understanding speech and audio · reviewed by AI
- When two audio clips are concatenated at the waveform level into one input, current large audio language models keep transcribing both segments nearly as well as single clips while their accuracy on content, order and location questions about the same two segments drops far below their single-clip attribute-recognition accuracy.Understanding speech and audio · reviewed by AI
- For precise anomaly localization in long multivariate time-series (16k-131k points), keeping the signal outside the model context as a variable the model queries through code over multiple turns recovers far more localization accuracy than single-pass ingestion, where the recording is serialized, subsampled or compressed into context once.Losing information in long inputs · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An evaluation of the same or later models on comparable timestamped audio showing purity at or above coverage, or TLoc F1 flat across the 0-30s to 120-180s duration bins.