arxiv-2609-21967 SRC-0509 · paper

NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities

Created: 2026-09-18 · Ingested: 2026-10-17

https://arxiv.org/abs/2609.21967

We introduce NemotronLabs VoiceChat, an open full-duplex speech-to-speech model with native tool-calling capabilities. NemotronLabs VoiceChat combines a streaming speech encoder and decoder-only language model with parallel specialized output streams for agent text and structured function calls, an auxiliary RNN-T branch for incremental user transcription, and a streaming TTS decoder. This design enables the model to listen, transcribe, reason, invoke tools, and speak within a unified streaming architecture while preserving the temporal behavior required for natural conversation. On Full-Duple

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsIn a full-duplex speech-to-speech model that emits function calls on a dedicated parallel output stream, choosing the right tool from spoken, disfluent user requests is far easier than filling its arguments: tool-selection F1 reached 82.5% (above two closed realtime APIs) while argument accuracy was 42.2% and end-to-end Pass@1 33.0%, below both.