AI Voice Agents: How They Work and Where They Fail

A voice AI agent listens to a caller, decides what to say or do, and speaks back, all in real time. It can be built in one of two ways. A cascade chains three models: speech recognition turns the audio into text, a language model decides the reply, and text to speech says it aloud. A speech-to-speech model takes audio in and produces audio out inside one network. How natural the call feels depends mostly on two things: how long the caller waits before hearing a reply, and whether the agent can tell that the caller has finished speaking. Common failures come from those two timing problems, from names and numbers that recognizers mishear, and from noise and interruptions. For the wider picture, read our overview of voice and AI.

A voice agent joins speech recognition, a language model and a voice in real time

A voice agent runs one loop each time the caller speaks: hear, decide, speak. In the cascade design, each step is a separate model, which makes each step easy to inspect. The first model is automatic speech recognition, the same family of technology behind AI dictation. It writes the caller's words as text, often while the caller is still talking. A large language model reads that text, calls any tools it needs, such as a booking system or an order lookup, and writes a reply. A text to speech model renders the reply as audio.

Dictation software is a different product. Yap, which we make, types the words you say into the app you are using. It does not answer a caller, so this guide covers only systems that talk back.

The diagram shows the cascade. OpenAI's developer guide describes the same choice as two paths. The chained path lets a developer inspect or change the text between the stages ¹. The other path uses one model that interprets the audio, decides what to do and responds in speech ².

The two kinds of product share the first step. What separates a voice agent from dictation is what happens next: the agent acts on the transcript instead of only writing it down. For the difference in practice, see voice assistants compared with dictation. The terms are defined in our glossary entries for voice agent and large language model.

Each stage of a cascade can come from a different vendor, which is why cascades are common in products that already use a preferred recognizer or voice. The cost is that every hand-off needs its own testing, and a fault in one stage often shows up as a symptom in another.

People expect a reply within a fraction of a second

Human conversation runs on short gaps. Across ten languages, the average time between one speaker finishing a turn and the next speaker starting was 208 milliseconds ⁴. An agent cannot match that gap exactly, but every stage of its loop adds time before the first word of the reply, so the budget has to be counted stage by stage.

The first stage is recognition. In Artificial Analysis's 2026 streaming benchmark, Deepgram Flux had the fastest final transcript after the end of speech, at 0.020 seconds ⁵. ElevenLabs Scribe v2 Realtime had the highest accuracy on the first partial transcript, which arrived 0.13 seconds after the end of speech ⁶. These are recognizer timings, not whole-call timings.

The second stage is the wait for silence, covered in the next section. In a streaming study by Sainath and colleagues, the endpointer alone had a median latency of 450 ms ⁷. The third stage is the language model, which must produce its first token before any speech can start. The fourth stage, the first audio of the voice, is not measured in the sources we checked, so test it on your own calls.

The numbers in the chart measure different things, so they show where the time goes rather than a total. In these figures, the wait for silence is longer than the recognizer's own delay. Our guide to real-time speech recognition explains the recognizer side in more depth.

Knowing when you have finished speaking is the hardest part

Turn detection decides when the caller has finished, and it is the step most likely to make an agent feel slow or rude. The usual method waits for silence. OpenAI's Realtime API uses periods of silence to split audio into turns by default ⁸. A shorter silence setting ends a turn sooner ⁹, which speeds up the reply but leaves less room for a pause mid-sentence.

The trade-off is real, and it falls unevenly across speakers. In a 2023 study of people who stutter, the baseline endpointer cut off an average of 23.8 percent of their utterances ¹⁰. Among people who stutter and were familiar with voice assistants, 61 percent named being cut off before they finished as a reason they used them less ¹¹.

Waiting longer prevents cut-offs but adds delay, so researchers try to make the detector smarter rather than simply slower. Some agents also add turn detection that reads the words as well as the silence, so a caller who pauses mid-sentence is less likely to be interrupted.

Some products avoid the problem by letting the speaker decide. The 2023 study notes that some smartphones let you hold a button while you speak and release it when you finish, which avoids automatic end-of-speech detection ¹².

Two practical rules follow. Set the silence wait from real call recordings, not from a demo. And test callers who pause often, because a single silence setting that suits fluent speakers may cut off others.

Speech-to-speech models cut delay but are harder to control

A speech-to-speech model removes the hand-offs between stages, and that is where its speed comes from. The research model Moshi, from the Kyutai lab, reports a theoretical latency of 160 ms and about 200 ms in practice ³. Those are the authors' own measurements in a research setup. They do not include phone networks, tool calls or the checks a commercial product adds, so they show what the design can do, not what a customer will hear.

The cost is control. When the text never appears as a separate step, you cannot easily check, filter or log it before it is spoken. The chained design keeps that step, and OpenAI's guide presents it as the way to inspect or transform text between recognition and speech ¹.

Pricing also differs by design. OpenAI lists its gpt-live-1 voice sessions at $0.05 per minute ¹³. Check whether each price is per minute or per token before comparing, and convert both to a cost for your own call length.

When you compare designs, ask a vendor for four things in writing: the time to the first audio of a reply on your kind of call, the silence setting used for turn detection, the price of a full minute including the model and the tools, and whether the call audio or transcript is stored. A demo rarely shows all four.

Vendor examples put a full agent near 8 to 13 cents a minute

A full agent costs more than its voice session, because the language model, the recognizer, the voice and the hosting are billed separately. Vapi's pricing page lists a pre-paid hosting fee of $0.05 per minute, with models billed at provider cost ¹⁴. Its example estimate for 1,000 minutes a month, using Deepgram transcription, OpenAI and ElevenLabs models, totals $82 to $129 ¹⁵. That works out to about 8.2 to 12.9 cents per minute ¹⁶.

Retell's pricing calculator gives a different split for a default setup: a $0.11 per minute total with an LLM cost, voice infrastructure and text to speech ¹⁷. Both examples set telephony at zero, so a real phone line adds its own carrier charge. Compare platforms only after you have matched the line items.

For a full cost walk-through, see our guide to AI dictation cost, which uses the same method for dictation, or the question on what an AI voice agent costs per minute.

Two things change the total in practice. Call length matters because every minute is billed. The language model is its own line in the Retell example, $0.04 of the $0.11 total ¹⁷, and its cost changes with the model you choose. Before you sign up, price a realistic call of your own length through every line of the estimate.

Voice agents fail on names, numbers, noise and interruptions

Many failures show up on the words that carry the action: a name, a street, an amount, or a reply that comes after a cut-off. An overall word error rate can look acceptable while those words fail. A 2026 study of 15 speech models from OpenAI, Deepgram, Google and Microsoft found an average transcription error rate of 44 percent on US street names spoken by US participants ¹⁸. A wrong street name matters most for any booking or dispatch agent that acts on an address.

Numbers and amounts need their own test. A general benchmark reports one overall rate, and it does not show how often a price or a date comes out wrong. Check those words on your own calls.

A practical way to do this is to log each call transcript and mark the words that carried the action, such as the name, the address, the amount and the appointment time. Count the errors on those words separately from the overall error rate, and set a threshold for each one before launch.

Benchmarks of whole conversations show the gap between text and voice. On τ-Voice, a benchmark of multi-turn tasks, GPT-5 with reasoning completed 85 percent of tasks, while voice agents reached 31 to 51 percent with clean audio ²³. With realistic audio, meaning noise, diverse accents and interruptions, voice agents reached 26 to 38 percent ²⁴. The authors also report that 79 to 90 percent of failures stem from agent behavior ²⁵. They present this as a result of their own setup, so treat it as a reason to test your agent's flow and tools as well as its speech.

Laws require AI callers to be disclosed and restrict automated calls

Two rules matter most for an agent that calls people. In the United States, an AI voice is treated as an artificial voice ²⁶, so the consent rules for automated calls apply. In the European Union, the agent must tell people they are interacting with an AI system ²⁷, unless that is obvious ²⁸. This section gives general information, not legal advice. Rules differ by country and by use, so check them with counsel before you deploy.

The FCC ruling clarifies existing rules rather than creating new ones ³², so the consent requirement is the same one that already covers robocalls. Disclosing that the caller is talking to an AI is a separate duty, and it does not replace consent. Keep a record of how consent was given, because that is the evidence a dispute turns on. Record the source of consent for every number you dial, and keep that record with the call log.

A checklist before you put an agent on the phone

1. Measure the full path on real recordings, from the end of the caller's speech to the first audio of the reply, not only the recognizer's first partial.
2. Test names, addresses and amounts as separate checks, because an overall word error rate hides them.
3. Test turn detection with pauses mid-sentence, including speakers who stutter or pause often.
4. Test noisy and distant audio, because the error rate rises quickly when the microphone is far from the caller.
5. Confirm consent and disclosure for every country and every type of call before the first call is placed.

Keep reading

• How fast should a voice agent respond?
• What is the difference between cascaded and speech-to-speech voice AI?
• Do AI voice agents have to disclose that they are AI?
• Speech to text and LLMs: what language models change
• Voice assistants vs dictation: Siri, Alexa, Gemini
• Word error rate, the standard accuracy measure
• Voice activity detection
• Speech-to-speech model
• Text to speech
• On-device speech recognition
• Speech recognition accuracy