audio-language-models-feed-encoder-outputs-into-text-llm CLM-0362In audio language models that feed encoder outputs into a text LLM, downstream understanding accuracy across speech, sound and music is set by how semantically text-aligned the encoder is rather than by whether its representation is discrete or continuous — reconstruction-oriented codecs with high acoustic fidelity underperform, semantically distilled tokenizers can beat their own continuous teacher, and scaling the LLM backbone does not recover the lost information.
Capability: Understanding speech and audio
Observed on
Encoders frozen or lightly adapted, fed through a projector into 135M-8B text LLM backbones (SmolLM2 with LoRA, Llama-3 frozen-probing); understanding tasks only, not generation or.
Sources
- Validated by: Nine encoders compared under identical task configs on ~20 datasets across speech/sound/music; Whisper led overall (0.735 vs 0.418 for Wav2Vec2 at 135M), SpeechTokenizer beat HuBERT on emotion recognition, DAC and WavTokenizer scored lowest on semantic tasks despite higher fidelity; performance fell in about 53% of encoder-task configs when the backbone was scaled.. Measured on one framework (UniARC) with nine encoders, four backbones, ~20 datasets, identical per-task configs. Projector is a simple MLP, which the authors note may itself bottleneck scaling; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In encoder-projector-decoder audio-language models, speaking style is linearly decodable from late audio-encoder layers at about 82-85% on a 7-way style task, yet open-ended tone prediction at the output reaches only about 20-54%, and which end of that range a model lands on tracks its training objective rather than its encoder: a model trained to generate speech keeps acoustic sensitivity through the decoder while text-only-trained siblings with a nearly identical encoder answer from the words instead.Understanding speech and audio · reviewed by AI
- On multi-session dialogue questions whose answers depend only on how an utterance was delivered, a cascaded ASR-then-LLM pipeline (Whisper large-v3 into a frontier model) scores below 7B audio-native models on the same clips, and below even a transcript-only baseline, because the transcription step drops delivery cues entirely and adds content errors.Understanding speech and audio · reviewed by AI
- For spoken language models answering questions over long dialogue histories, converting the history to transcripts beats keeping it as audio when the answer is a lexical detail (exact slot values across dense turns: mean 71.3% vs 42.6%), but loses when the answer depends on how something was said, where audio history wins for six of seven models and by a larger margin once lexical emotion cues are removed.Understanding speech and audio · reviewed by AI
- When post-training a speech LLM for ASR with GRPO, adding a reward term from a separately pretrained, frozen CTC acoustic judge that scores each sampled transcription against the audio cuts insertion errors on out-of-distribution meeting speech (22-28% fewer on AMI) relative to a schedule-matched text-only -WER reward, without raising word error rate and without needing the judge at inference.Stating false facts confidently · reviewed by AI
- For large audio language models on multiple-choice audio reasoning benchmarks (MMAR, MMAU-mini), accuracy does not track whether the model's chain of thought is acoustically grounded or causally drives the answer: several models answer correctly from perception steps that assert unsupported labels, and for some models replacing the reasoning chain with vacuous or fabricated content leaves the final answer unchanged.Understanding speech and audio · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A controlled comparison in which a purely reconstruction-trained codec matches or beats text-aligned encoders on speech/sound/music understanding under the same projector and backbone, or in which scaling the backbone closes the gap between low- and high-semantic encoders on data-limited tasks.