Glossary
Speech-to-speech model
A speech-to-speech model takes audio in and produces audio out with one model, without writing the conversation out as text in between. Most voice systems instead chain speech recognition, a language model and speech synthesis.
Updated October 17, 20264 cited facts, each traced to its source
Also called end-to-end voice model, realtime voice model, S2S.
A speech-to-speech model (also called an end-to-end voice model, realtime voice model or S2S) is one neural network that listens to speech and generates speech in reply. It does not need to produce a transcript of your words first. The usual alternative is a cascade: speech recognition writes your words as text, a language model writes a reply as text, and speech synthesis reads that reply aloud.
Kyutai's paper on Moshi says that current systems for spoken dialogue rely on pipelines of independent components, namely voice activity detection, speech recognition, textual dialogue and text-to-speech. It then casts spoken dialogue as speech-to-speech generation as the alternative 1.
Why a cascade is slow and lossy
Each stage in a cascade waits for the one before it. The Moshi paper says that this complexity induces a latency of several seconds between interactions 2. The same paper says that when text is the intermediate step, non-linguistic information that changes meaning is lost 3. Those are the two problems a single model is meant to solve.
What the latency numbers show
Moshi, a full-duplex speech-to-speech model from Kyutai, has a theoretical latency of 160 ms and about 200 ms in practice 4. These are the authors' own measurements in a research setup. They describe the model, not a phone app or a call center product, so treat them as an upper bound on what a deployed system will match.
Why it matters for dictation
Dictation needs text, because the text is the product: you read it, edit it and paste it into a document. A speech-to-speech model answers in audio, so it is a poor fit for putting your own words on the page. Most dictation tools produce a transcript as their output.
A speech-to-speech design also removes a checkpoint you may rely on. In a cascade, the recognized text is something you can inspect and correct before a reply is made. A single model without a visible text stage gives you less to check. When you choose a tool for writing, look for a transcript you can edit, and use speech-to-speech systems for conversation instead.
Related terms
- Voice agent: a conversational system that may be built on a cascade or on a speech-to-speech model.
- Speech language model: a language model that accepts speech input directly.
- Automatic speech recognition: the first stage of a cascade, which writes speech as text.
- Text to speech: the final stage of a cascade, which reads text aloud.
- Turn detection: deciding when the speaker has finished, which both designs need.
Guides on this topic
Evidence
Every number on this page, with where it comes from. Open a fact to see the exact words in its source.
- 1.
- 2.
- 3.
- 4.
Related terms
- Speech to Text AI: Dictation, Assistants and Voice AgentsThe topic this term belongs to
- AI text detectorAn AI text detector is software that estimates whether a passage was written by a language model, usually from the words alone. Detectors make mistakes in both directions, so a score is weak evidence about how a text was made.
- Generative AIGenerative AI is the class of AI models that create new content, such as text, audio or images, from patterns in their training data. The NIST profile defines it as models that emulate the structure and characteristics of input data to generate derived synthetic content.
- Generative error correction (GER)Generative error correction (GER) passes a speech recognizer's output, often its N-best list of alternative transcripts, to a large language model, which rewrites the transcript to fix errors. It can recover words the recognizer never offered, but it can also change words that were already right.
- Intent recognitionIntent recognition is the step that decides what a spoken request is asking for, such as setting a timer or sending a message, and which details the action needs. Voice assistants and voice agents rely on it before they act.
- Large language model (LLM)A large language model (LLM) is a language model with a very large number of parameters, trained on huge amounts of text to predict and generate language. Google's glossary defines the minimum as a language model having a very high number of parameters.
- Machine learningMachine learning is a method in which a program learns its behavior from example data rather than from rules written by hand. Google's glossary defines it as a program or system that trains a model from input data.
- Prompt (AI)In AI, a prompt is the input you give a model to get a response, such as a question, an instruction or pasted text. Google's glossary defines it as any text entered as input to a large language model to condition the model to behave in a certain way.
- Speech language modelA speech language model (speech LLM) is a large language model that accepts audio as input, so it can process speech directly instead of working only on a text transcript. The 2023 SALMONN model is one example that lets an LLM process general audio inputs.
- Text to speech (TTS)Text to speech (TTS), also called speech synthesis, is software that converts written text into spoken audio. Modern systems use neural networks, and they can run on the device or in the cloud.
- Turn detectionTurn detection is the step that decides when a speaker has finished a turn, so a voice system knows when to respond. A pause can be a hesitation rather than an ending, so the decision trades speed against cutting the speaker off.
- Voice agentA voice agent is a voice system that listens, reasons with a language model, uses tools, and speaks back across several turns to finish a task, such as checking an order and rescheduling it. Benchmarks show that agents still fail many tasks on realistic audio.
- Voice assistantA voice assistant is software that listens for a spoken request, works out what you want, and then acts or answers, such as setting a timer or turning off a light. Dictation is different: it turns your speech into text for you to edit.
Pages that use this term
- Speech to Text AI: Dictation, Assistants and Voice AgentsTopic
- AI Voice Agents: How They Work and Where They FailGuide
- Can Whisper AI do text to speech?Question
- How do you connect speech to text to an LLM?Question
- What is the difference between cascaded and speech-to-speech voice AI?Question
- Speech language modelGlossary
- Text to speech (TTS)Glossary
- Voice agentGlossary