How fast does a voice AI agent need to respond?

A voice agent should start speaking within about half a second after the caller stops. That is a target we recommend, not a published standard. Keep the silence well under one second, because past that point the call loses its rhythm. People leave an average of 208 milliseconds between their turns ¹, so an agent that waits longer is noticed.

Aim for a reply that starts within half a second

Conversation runs on short gaps. Across ten languages, the average time between one speaker finishing and the next starting was 208 milliseconds ¹. The same research found that speakers react to small timing changes. They become hypersensitive to changes in the timing of responses of 100 milliseconds or less ². An agent cannot match a human gap every time, so a reply that begins within about half a second is a sensible goal.

Beyond about a second, the call loses its rhythm. Jakob Nielsen's usability research gives 1.0 second as about the limit for a user's flow of thought to stay uninterrupted ³. That rule was written for screens, not phone calls, so treat it as a rough check. If a caller sits in silence for a second or more, they may start talking again, and the agent then has to handle an interruption.

Count every stage, not only the recognizer

The delay adds up across the parts of the loop. Recognition is only one of them. In a streaming study by Sainath and colleagues, the endpointer alone, the part that decides the speaker has stopped, had a median latency of 450 milliseconds ⁴. That is longer than the whole human gap, and it comes before the language model writes its first word and the voice starts speaking.

So a reply that begins in under a second needs every stage to be quick: the endpointer, the model's first token, and the first audio of the voice. Vendor benchmarks often report one of these numbers alone. Ask for the time from the end of the caller's speech to the first audio of the reply, measured on calls like yours.

Turn detection is the stage you can tune most directly

Turn detection is the step that decides the caller has finished. OpenAI's Realtime API uses periods of silence to split audio into turns by default, for models that support its server-side detection ⁵. A shorter silence setting ends a turn sooner, so the reply can start sooner ⁶. The trade-off is that a short setting also cuts callers off. In a 2023 study of people who stutter, the baseline endpointer cut off an average of 23.8 percent of their utterances ⁷. Among people who stutter and were familiar with voice assistants, 61 percent named being cut off as a reason they used them less ⁸.

For more on the step, see the glossary entries for turn detection and voice activity detection.

It depends: some replies can take longer

Speed matters most on turns where the caller expects an immediate answer, such as a greeting, a yes or no, or a confirmation. A reply that needs a booking lookup or a payment check will take longer, and the caller can tolerate that when they know the agent is still working. The fix is a design choice, not a faster model: confirm what is happening, then give the answer. Test that choice on recorded calls before you rely on it.

The right silence setting also depends on who calls. A setting that suits fluent speakers can cut off people who pause often, so a single fixed value fails some callers. Test with pauses in the middle of sentences, not only with clean demo speech.

The design you choose sets the floor on speed. A cascade of separate models adds its stages, while a speech-to-speech model removes some hand-offs. The cascaded and speech-to-speech comparison explains that choice.

What to do

1. Set a target for the first word of the reply, measured from the end of the caller's speech.
2. Log each stage separately: recognition, turn detection, the model's first token and the voice's first audio.
3. Tune the silence setting on recordings that include slow speakers and people who pause often.
4. Test the slow turns, such as lookups and payments, and decide what the caller hears while the agent works.
5. Measure again after every change to the model, the voice or the vendor.

The overview of the subject is the guide to AI voice agents, how they work and where they fail, and the wider context is in our voice and AI overview.