What is the difference between cascaded and speech-to-speech voice AI?

The two designs differ in where the text sits. A cascaded voice AI turns the caller's speech into text, decides a reply as text, and turns that text back into speech. A speech-to-speech model takes audio in and gives audio out inside one model, with no separate text step. The cascade gives you control over the words. Speech to speech gives you speed and tone, at the cost of that control.

A cascade chains three models and keeps the text in between

A cascaded voice AI runs the caller's audio through a speech recognizer, which writes it as text. A large language model reads that text and decides the reply, then a text to speech model says it aloud. Because each hand-off produces text, a developer can log it, check it, filter it or change it before the caller hears anything.

OpenAI says the chained path is the one to use when you want to inspect or change the text between speech recognition, the agent and speech generation ¹. That is why cascades suit products that must follow scripts or check policy. Each stage can also come from a different vendor, which is why a team can keep a recognizer it trusts and replace only the voice.

The cost is testing. Every hand-off needs its own check, and a fault in one stage often shows up as a symptom in another.

A speech-to-speech model hears and speaks in one model

A speech-to-speech model takes audio in and produces audio out. OpenAI describes this path as a single model that takes audio in, decides what to do and speaks back ². With no separate text step to finish first, the reply can begin sooner.

The research model Moshi, from the Kyutai lab, reports a theoretical latency of 160 ms and about 200 ms in practice ³. Those are the authors' figures from a research setup, so read them as the best case for the design, not a number a caller will hear.

Because the reply is generated as audio, a speech-to-speech model can carry tone and emphasis that plain text would lose. That matters most in casual conversation. It matters less when the job is to read back an exact address or price.

The trade-off is speed and tone against control

The table summarizes the pattern. Your own calls can break it. A cascade with a fast recognizer and a short silence setting can feel quicker than a slow speech-to-speech model, so measure both on the same recordings. The speed guide explains which delays to count.

Pricing and vendor claims differ by design

A speech-to-speech session can be quoted as one rate, while a cascade is priced part by part. OpenAI lists its gpt-live-1 voice sessions at $0.05 per minute ⁴. Ask whether the backend model and tool usage are billed on top of that rate. For a cascade, add the recognizer, the language model and the voice, then add the cost of the tools the agent calls. Compare the full cost per minute of each design.

The cost question shows how the lines add up on a phone call.

It depends on the call

Choose by what must be checked before the caller hears a reply.

• Scripted, regulated or high-stakes calls. Booking changes, payment steps and any call with a required disclosure benefit from a text step you can log and check. A cascade fits well here.
• Casual conversation and low-latency assistants. When the goal is a natural back-and-forth and the words matter less than the rhythm, a speech-to-speech model is the stronger candidate.
• Tool-heavy work. In a cascade, the model's tool calls appear as text you can log, which helps debugging.
• Fast-moving stacks. A cascade lets you replace one component when a better one appears. A speech-to-speech model usually means replacing the whole model.

What to do before you choose

1. List the call types and write down what must be checked before the caller hears a reply.
2. Choose the cascade if you need a transcript or a check on the text. Choose speech to speech if rhythm and tone come first.
3. Measure both designs on the same recordings, timing the first audio of the reply.
4. Ask each vendor for the price of a full minute, including the model, the tools and any telephony.

For the wider subject, read the guide to AI voice agents, how they work and where they fail. The terms are defined in our glossary entries for voice agent, speech-to-speech model and large language model. For the wider subject, see the voice and AI overview.