llm-based-asr-steering-decoding-toward-acoustic-evidence-only-helps CLM-0078In LLM-based ASR, steering decoding toward acoustic evidence only helps if candidates are first restricted to tokens within a small likelihood margin of the greedy token: unconstrained selection by an acoustic compatibility score removes hallucinated outputs but destroys transcription accuracy through unbounded insertions, while the constrained version removes a large share of hallucinations at near-zero WER/CER cost.
Capability: Stating false facts confidently
Observed on
Four open LLM-based ASR systems (Qwen3-ASR-1.7B, Qwen2-Audio-7B Base, Kimi-Audio-7B-Instruct, GLM-ASR-Nano) on two 500-utterance stress suites built to elicit code-switching and in.
Sources
- Paired greedy-vs-LCAR decoding on 8800 frozen detector-positive records plus negative controls, with an LLM detector whose precision was human-audited at 93.3% on a 400-case stratified sample. Ablation on 800 Qwen3 positives isolates the likelihood constraint: acoustic- only selection gives comparable overall hallucination fixing but >600/1000 point WER/CER increases. Normal-speech effects measured on LibriSpeech test-clean and AISHELL2. Gains vary by model and suite; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On multi-session dialogue questions whose answers depend only on how an utterance was delivered, a cascaded ASR-then-LLM pipeline (Whisper large-v3 into a frontier model) scores below 7B audio-native models on the same clips, and below even a transcript-only baseline, because the transcription step drops delivery cues entirely and adds content errors.Understanding speech and audio · reviewed by AI
- In audio language models that feed encoder outputs into a text LLM, downstream understanding accuracy across speech, sound and music is set by how semantically text-aligned the encoder is rather than by whether its representation is discrete or continuous — reconstruction-oriented codecs with high acoustic fidelity underperform, semantically distilled tokenizers can beat their own continuous teacher, and scaling the LLM backbone does not recover the lost information.Understanding speech and audio · reviewed by AI
- When post-training a speech LLM for ASR with GRPO, adding a reward term from a separately pretrained, frozen CTC acoustic judge that scores each sampled transcription against the audio cuts insertion errors on out-of-distribution meeting speech (22-28% fewer on AMI) relative to a schedule-matched text-only -WER reward, without raising word error rate and without needing the judge at inference.Stating false facts confidently · reviewed by AI
- For encoder-based hallucination/claim-faithfulness detectors moved from general-domain data to biomedical claim verification, which corpus the encoder was pre-trained on matters more than which data it is fine-tuned on: a PubMed-pretrained encoder fine-tuned on a few hundred SciFact pairs beat both a general encoder fine-tuned on the same pairs and a general encoder pre-fine-tuned on a large general hallucination set.Competence in a specialist domain · reviewed by AI
- In encoder-projector-decoder audio-language models, speaking style is linearly decodable from late audio-encoder layers at about 82-85% on a 7-way style task, yet open-ended tone prediction at the output reaches only about 20-54%, and which end of that range a model lands on tracks its training objective rather than its encoder: a model trained to generate speech keeps acoustic sensitivity through the decoder while text-only-trained siblings with a nearly identical encoder answer from the words instead.Understanding speech and audio · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An unconstrained acoustic-reranking decoder that matches or beats the likelihood-constrained version on both hallucination removal and standard-test WER/CER, or a constrained variant whose hallucination reduction comes with large WER regressions on normal speech. Proposed technique, not catalogued: likelihood-constrained acoustic reranking.