Glossary
Voice agent
A voice agent is a voice system that listens, reasons with a language model, uses tools, and speaks back across several turns to finish a task, such as checking an order and rescheduling it. Benchmarks show that agents still fail many tasks on realistic audio.
Updated October 17, 20264 cited facts, each traced to its source
Also called AI voice agent, conversational voice AI, voicebot.
A voice agent (also called an AI voice agent, conversational voice AI or voicebot) is a voice system that holds a spoken conversation and carries out a task on your behalf. It listens, uses a language model to decide what to do next, calls tools such as a booking system or an order database, and speaks back. The job is finished when the outcome is done, not when a single question is answered.
The research framing is consistent. The VoiceAgentBench paper says large speech language models have enabled voice assistants that understand natural spoken queries and perform complex tasks 1. Its authors go on to argue that the usual speech tests miss the agent part of the job.
How a voice agent differs from a voice assistant
A voice assistant typically handles one request and stops. A voice agent plans. It may call several tools in order, keep track of what was said earlier in the call, and ask for confirmation before it changes anything. The VoiceAgentBench authors describe their benchmark as covering single-tool calls, multi-tool workflows, multi-turn dialogue and safety tests, all with spoken queries ( VoiceAgentBench abstract).
The gap they point to is real. The paper says existing speech benchmarks largely focus on isolated capabilities such as transcription or question answering, and do not systematically evaluate agentic behavior or adversarial robustness 2.
Where voice agents still fail
Task success is much lower on spoken input than on text. In the τ-Voice benchmark, GPT-5 with reasoning completed 85 percent of tasks, against 31 to 51 percent for voice agents on clean audio 3. Failure in a voice agent is hard to spot from the transcript alone. A call can sound fluent and still book the wrong slot or read back the wrong number. That is why confirmation steps matter more for agents than for dictation.
Why it matters for dictation
Dictation writes your words. An agent acts on them, and an action can be hard to undo. Apple's Siri on iPhone shows a safer pattern: when you dictate a text with Siri, it reads the text back and waits for you to confirm before sending 4. When a tool offers to act on your speech, look for that step. If it does not show you what it is about to do, keep it out of anything you cannot easily take back.
Our guide to AI voice agents covers how these systems are built and where they fail in real calls, and the overview of speech to text AI explains where agents sit next to dictation.
Related terms
- Voice assistant: the simpler system that agents extend.
- Intent recognition: the step that decides which tool or action a request needs.
- Speech-to-speech model: a single model that can answer in audio without a text stage.
- Turn detection: deciding when the caller has finished speaking so the agent can reply.
Guides on this topic
Evidence
Every number on this page, with where it comes from. Open a fact to see the exact words in its source.
- 1.
- 2.
- 3.
- 4.
Related terms
- Speech to Text AI: Dictation, Assistants and Voice AgentsThe topic this term belongs to
- AI text detectorAn AI text detector is software that estimates whether a passage was written by a language model, usually from the words alone. Detectors make mistakes in both directions, so a score is weak evidence about how a text was made.
- Generative AIGenerative AI is the class of AI models that create new content, such as text, audio or images, from patterns in their training data. The NIST profile defines it as models that emulate the structure and characteristics of input data to generate derived synthetic content.
- Generative error correction (GER)Generative error correction (GER) passes a speech recognizer's output, often its N-best list of alternative transcripts, to a large language model, which rewrites the transcript to fix errors. It can recover words the recognizer never offered, but it can also change words that were already right.
- Intent recognitionIntent recognition is the step that decides what a spoken request is asking for, such as setting a timer or sending a message, and which details the action needs. Voice assistants and voice agents rely on it before they act.
- Large language model (LLM)A large language model (LLM) is a language model with a very large number of parameters, trained on huge amounts of text to predict and generate language. Google's glossary defines the minimum as a language model having a very high number of parameters.
- Machine learningMachine learning is a method in which a program learns its behavior from example data rather than from rules written by hand. Google's glossary defines it as a program or system that trains a model from input data.
- Prompt (AI)In AI, a prompt is the input you give a model to get a response, such as a question, an instruction or pasted text. Google's glossary defines it as any text entered as input to a large language model to condition the model to behave in a certain way.
- Speech language modelA speech language model (speech LLM) is a large language model that accepts audio as input, so it can process speech directly instead of working only on a text transcript. The 2023 SALMONN model is one example that lets an LLM process general audio inputs.
- Speech-to-speech modelA speech-to-speech model takes audio in and produces audio out with one model, without writing the conversation out as text in between. Most voice systems instead chain speech recognition, a language model and speech synthesis.
- Text to speech (TTS)Text to speech (TTS), also called speech synthesis, is software that converts written text into spoken audio. Modern systems use neural networks, and they can run on the device or in the cloud.
- Turn detectionTurn detection is the step that decides when a speaker has finished a turn, so a voice system knows when to respond. A pause can be a hesitation rather than an ending, so the decision trades speed against cutting the speaker off.
- Voice assistantA voice assistant is software that listens for a spoken request, works out what you want, and then acts or answers, such as setting a timer or turning off a light. Dictation is different: it turns your speech into text for you to edit.
Pages that use this term
- Speech to Text AI: Dictation, Assistants and Voice AgentsTopic
- AI Voice Agents: How They Work and Where They FailGuide
- Do AI voice agents have to disclose that they are AI?Question
- How much does an AI voice agent cost per minute?Question
- What is the difference between cascaded and speech-to-speech voice AI?Question
- Intent recognitionGlossary
- Speech language modelGlossary
- Speech-to-speech modelGlossary
- Text to speech (TTS)Glossary
- Turn detectionGlossary
- Voice assistantGlossary