full-duplex-speech-to-speech-model-emits-function-calls-dedicated-parallel-output CLM-0396observationsingle paperReviewed by AI
In a full-duplex speech-to-speech model that emits function calls on a dedicated parallel output stream, choosing the right tool from spoken, disfluent user requests is far easier than filling its arguments: tool-selection F1 reached 82.5% (above two closed realtime APIs) while argument accuracy was 42.2% and end-to-end Pass@1 33.0%, below both.
Capability: Using the tools it is given
Observed on
One open 9B-backbone speech-to-speech model evaluated on Full-Duplex-Bench 3.0 (real human speech, chained API calls); authors also report degradation beyond about five exposed too.
Sources
- supportsNemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling CapabilitiesValidated by: Full-Duplex-Bench 3.0 tool-selection F1, argument accuracy and Pass@1 against Gemini Live 2.5 and 3.1; full-duplex behaviour checked on FDB 1.0/1.5 and VoiceBench, but no ablation against a serialized single-channel variant.. Measured on one benchmark with one model; baselines are two closed APIs whose numbers come from the benchmark paper. No ablation isolating the parallel-stream design from the backbone or training mixture, so the architecture's contribution to the selection score is not established.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On multi-session dialogue questions whose answers depend only on how an utterance was delivered, a cascaded ASR-then-LLM pipeline (Whisper large-v3 into a frontier model) scores below 7B audio-native models on the same clips, and below even a transcript-only baseline, because the transcription step drops delivery cues entirely and adds content errors.Understanding speech and audio · reviewed by AI
- When post-training a speech LLM for ASR with GRPO, adding a reward term from a separately pretrained, frozen CTC acoustic judge that scores each sampled transcription against the audio cuts insertion errors on out-of-distribution meeting speech (22-28% fewer on AMI) relative to a schedule-matched text-only -WER reward, without raising word error rate and without needing the judge at inference.Stating false facts confidently · reviewed by AI
- When large audio language models are asked to return every time span matching a natural-language query, they almost always report fewer intervals than exist: across 21 systems on queries averaging about 4 ground-truth intervals, under-reporting rates run from 79% to 100% and no system exceeds 13.2% exact count accuracy, even for the models with the best overlap scores.Understanding speech and audio · reviewed by AI
- For spoken language models answering questions over long dialogue histories, converting the history to transcripts beats keeping it as audio when the answer is a lexical detail (exact slot values across dense turns: mean 71.3% vs 42.6%), but loses when the answer depends on how something was said, where audio history wins for six of seven models and by a larger margin once lexical emotion cues are removed.Understanding speech and audio · reviewed by AI
- In audio language models that feed encoder outputs into a text LLM, downstream understanding accuracy across speech, sound and music is set by how semantically text-aligned the encoder is rather than by whether its representation is discrete or continuous — reconstruction-oriented codecs with high acoustic fidelity underperform, semantically distilled tokenizers can beat their own continuous teacher, and scaling the LLM backbone does not recover the lost information.Understanding speech and audio · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A speech-to-speech model on the same benchmark whose argument accuracy and Pass@1 match or exceed its tool-selection F1, or closed APIs showing the same selection-over-argument gap in reverse.