Understanding speech and audio

Claims

• observationsingle paper↑1↓0
  In audio in-context learning with per-episode resampled rules, current large audio language models bind arbitrary non-speech sounds to new labels reasonably well (best model 58.5% average over six such tasks at 5 shots) but stay near zero on tasks requiring temporal measurement — duration estimation, onset/offset segmentation, Morse dot-versus-dash — with no model averaging above 7.1%, and that temporal gap does not shrink across model generations while the non-temporal one improves sharply.
• mechanismsingle paper↑1↓0
  In audio language models that feed encoder outputs into a text LLM, downstream understanding accuracy across speech, sound and music is set by how semantically text-aligned the encoder is rather than by whether its representation is discrete or continuous — reconstruction-oriented codecs with high acoustic fidelity underperform, semantically distilled tokenizers can beat their own continuous teacher, and scaling the LLM backbone does not recover the lost information.
• observationsingle paper↑1↓0
  On audio temporal-localization tasks in 11 Southeast Asian languages, current audio LLMs return time spans whose coverage of the true span consistently exceeds their purity, and F1 falls steadily as the recording grows from under 30 seconds to 3 minutes, with several models unable to run at all on the longer bins.
• mechanismsingle paper↑1↓0
  In encoder-projector-decoder audio-language models, speaking style is linearly decodable from late audio-encoder layers at about 82-85% on a 7-way style task, yet open-ended tone prediction at the output reaches only about 20-54%, and which end of that range a model lands on tracks its training objective rather than its encoder: a model trained to generate speech keeps acoustic sensitivity through the decoder while text-only-trained siblings with a nearly identical encoder answer from the words instead.
• observationsingle paper↑1↓0
  When large audio language models are asked to return every time span matching a natural-language query, they almost always report fewer intervals than exist: across 21 systems on queries averaging about 4 ground-truth intervals, under-reporting rates run from 79% to 100% and no system exceeds 13.2% exact count accuracy, even for the models with the best overlap scores.
• observationsingle paper↑1↓0
  For large audio language models on multiple-choice audio reasoning benchmarks (MMAR, MMAU-mini), accuracy does not track whether the model's chain of thought is acoustically grounded or causally drives the answer: several models answer correctly from perception steps that assert unsupported labels, and for some models replacing the reasoning chain with vacuous or fabricated content leaves the final answer unchanged.
• mechanismsingle paper↑1↓0
  On multi-session dialogue questions whose answers depend only on how an utterance was delivered, a cascaded ASR-then-LLM pipeline (Whisper large-v3 into a frontier model) scores below 7B audio-native models on the same clips, and below even a transcript-only baseline, because the transcription step drops delivery cues entirely and adds content errors.
• observationsingle paper↑1↓0
  On multiple-choice audio questions where each answer choice is paired with a generated image, adding the images helps native multimodal models most on environmental-sound questions and not at all on speech (speech is best audio-only), and in cascaded pipelines that convert audio to transcripts or captions first the images lower average accuracy.
• observationsingle paper↑1↓0
  For spoken language models answering questions over long dialogue histories, converting the history to transcripts beats keeping it as audio when the answer is a lexical detail (exact slot values across dense turns: mean 71.3% vs 42.6%), but loses when the answer depends on how something was said, where audio history wins for six of seven models and by a larger margin once lexical emotion cues are removed.
• observationsingle paper↑1↓0
  When two audio clips are concatenated at the waveform level into one input, current large audio language models keep transcribing both segments nearly as well as single clips while their accuracy on content, order and location questions about the same two segments drops far below their single-clip attribute-recognition accuracy.

Techniques

• Pick audio encoders by semantic alignment, not fidelityarchitecture
  When choosing an audio front end for an audio-LLM, select encoders whose training imposed text/semantic constraints (weakly supervised or semantically distilled) instead of maximizing reconstruction fidelity or preferring continuous over di
• Pick history modality by what the question needsprompting
  Feed long dialogue history as transcripts for lexical-recall questions and as raw audio for paralinguistic questions, rather than one format for all.
• Score audio CoTs on grounding, coherence and causalityprocess
  Prompts the model with a four-step Perception/Analysis/Reasoning/Answer template, then scores the chain with six automatic metrics (CLAP audio-entity grounding, inter-step embedding coherence, small-LM substantiveness, perplexity-based step
• Score occurrence count separately from boundary overlapprocess
  Report, alongside union-IoU, the share of queries where the number of predicted intervals equals the number of ground-truth intervals, plus under- and over-reporting rates, so missing occurrences is distinguished from imprecise boundaries.

Sources

Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.

• Vox-Infinity: Benchmarking the Limits of Long-Context Spoken Language Models2026-09-21
  ↑ supports a claim here
• Discrete vs. Continuous: A Comprehensive Study of Unified Audio Understanding in LALMs2026-09-19
  ↑ supports a claim here
• AudioICL-Bench: A Benchmark for Large Audio Language Model In-Context Learning2026-09-10
  ↑ supports a claim here
• SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia2026-09-09
  ↑ supports a claim here
• Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models2026-09-09
  ↑ supports a claim here
• Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models2026-09-01
  ↑ supports a claim here
• TAG-Bench: Benchmarking Temporal Audio Grounding in Large Audio Language Models2026-09-01
  ↑ supports a claim here
• VoiceLongMemEval: Do Assistants Remember How You Sounded?2026-09-01
  ↑ supports a claim here
• Direct or Mediated? Task-Dependent Audio Information Routing in Large Audio Language Models2026-08-27
  ↑ supports a claim here
• EXAM$^2$: $\underline{Ex}tending$ $\underline{A}udio$ $Understanding$ $in$ $\underline{M}ultilingual$ $and$ $\underline{M}ultimodal$ $Analysis$2026-08-24
  ↑ supports a claim here
• HyPASE: Hyperbolic Geometry for Parameter-Efficient Speech Emotion Fine-Tuning Framework for Large Audio-Language Models2026-08-05
  about this capability · no claim drafted yet