Preprint
VoiceAgentBench: Are Voice Assistants ready for agentic tasks?
Dhruv Jain et al., arXiv, 2025
Read the originalRead on October 17, 2026
- What it is
- A 2025 benchmark of voice assistants on agentic tasks such as tool calling, comparing end-to-end speech language models with pipelines that transcribe speech first and then pass the text to a language model. Appendix F measures the time to first token for both designs on one GPU.
- How much weight it carries
- A preprint from Oct 2025. The latency numbers come from a single hardware setup (one H100 GPU) and 100 queries, so they show the size of the gap rather than a fixed delay for any product. The paper reports 90th percentile times, which are the delays that users notice most.
Facts taken from this source
- Qwen2.5-Omni 7B, an end-to-end speech language model, had a 90th percentile time to first token of about 40 milliseconds on one H100 GPU.
“Qwen2.5-Omni 7B achieves a 90th percentile (p90) TTFT of approximately 40 ms on a single H100 GPU” (Appendix F)
- A pipeline that runs Whisper large v3 first and then the Qwen3 8B language model had a 90th percentile time to first token of about 800 milliseconds on the same hardware.
“a pipeline combining Whisper-large-v3 with Qwen3 8B exhibits a p90 TTFT of around 800 ms under the same hardware conditions” (Appendix F)
- The VoiceAgentBench paper says existing speech benchmarks mostly test isolated skills such as transcription or question answering, and do not systematically test agentic behavior or adversarial robustness.
“existing speech benchmarks largely focus on isolated capabilities such as transcription or question answering and do not systematically evaluate agentic behavior or adversarial robustness.” (Abstract, second sentence)
- The VoiceAgentBench paper says large speech language models have enabled voice assistants that understand natural spoken queries and perform complex tasks.
“Large scale Speech Language Models have enabled voice assistants capable of understanding natural spoken queries and performing complex tasks.” (Abstract, first sentence)