speech-llms-long-audio-text-centric-kv-cache-eviction-methods CLM-0641For speech LLMs on long audio, text-centric KV cache eviction methods (StreamingLLM, SnapKV, PyramidKV, ChunkKV) collapse because raw audio tokens have diffuse attention; anchoring compression on ASR transcripts — pooling each transcribed audio span down to its text-token length and adding the text embedding, while keeping untranscribed audio intact — retains over 90% of full-cache accuracy at a 5-10% cache budget where those baselines retain roughly 55-67% at a looser 25% budget.
Capability: Losing information in long inputs
Observed on
Qwen3-Omni-30B (Instruct and Thinking) on six audio benchmarks; needs an auxiliary ASR model with timestamps and confidence scores; degrades with noisy or accented speech..
Sources
- Validated by: Accuracy against a Full-KV baseline and four text-domain KV compression baselines across Vox-Infinity, AudioMarathon, SPIRAL-Hard, MMAR, MMSU, MMAU, plus ablations removing semantic anchoring, removing untranscribed segments, text-only input, and sweeping the decay factor; efficiency measured at 64K context including ASR overhead.. Measured on one backbone family (Qwen3-Omni-30B) across six benchmarks with four compression baselines; ablations isolate the anchoring stage (PM subset 6.4% without vs 81.6% with at 15% budget) and the decay factor. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On multi-session dialogue questions whose answers depend only on how an utterance was delivered, a cascaded ASR-then-LLM pipeline (Whisper large-v3 into a frontier model) scores below 7B audio-native models on the same clips, and below even a transcript-only baseline, because the transcription step drops delivery cues entirely and adds content errors.Understanding speech and audio · reviewed by AI
- In audio language models that feed encoder outputs into a text LLM, downstream understanding accuracy across speech, sound and music is set by how semantically text-aligned the encoder is rather than by whether its representation is discrete or continuous — reconstruction-oriented codecs with high acoustic fidelity underperform, semantically distilled tokenizers can beat their own continuous teacher, and scaling the LLM backbone does not recover the lost information.Understanding speech and audio · reviewed by AI
- When post-training a speech LLM for ASR with GRPO, adding a reward term from a separately pretrained, frozen CTC acoustic judge that scores each sampled transcription against the audio cuts insertion errors on out-of-distribution meeting speech (22-28% fewer on AMI) relative to a schedule-matched text-only -WER reward, without raising word error rate and without needing the judge at inference.Stating false facts confidently · reviewed by AI
- For an 8B Llama backbone under a fixed 256-slot cache budget, replacing hard KV eviction with a learned soft write that blends a new token's key/value into a routed fixed-size memory slot retains far more long-context retrieval accuracy than sliding-window or sink caches, and unlike full-attention or quantized caches it does not run out of memory at 128K tokens.Losing information in long inputs · reviewed by AI
- In LLM-based ASR, steering decoding toward acoustic evidence only helps if candidates are first restricted to tokens within a small likelihood margin of the greedy token: unconstrained selection by an acoustic compatibility score removes hallucinated outputs but destroys transcription accuracy through unbounded insertions, while the constrained version removes a large share of hallucinations at near-zero WER/CER cost.Stating false facts confidently · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: On another speech LLM backbone, ASR-anchored compression retains no more of the full-cache score at a 5-20x budget than SnapKV/PyramidKV, or the semantic-anchoring ablation shows the gain comes from the decay rule alone.