Glossary
Speech language model
A speech language model (speech LLM) is a large language model that accepts audio as input, so it can process speech directly instead of working only on a text transcript. The 2023 SALMONN model is one example that lets an LLM process general audio inputs.
Updated October 17, 20263 cited facts, each traced to its source
Also called speech LLM, audio language model, speech-aware LLM.
A speech language model (speech LLM) is a large language model that takes audio as an input. It works on the sound itself rather than on a transcript, so it can answer a spoken question, transcribe a recording or translate it. The SALMONN authors say their model lets a large language model process and understand general audio inputs directly, with competitive performance on speech tasks such as automatic speech recognition 1.
Two designs: a cascade or one model
Speech systems that use an LLM come in two broad designs. The older design is a cascade, in which a speech recognizer writes text and a language model then works on that text. The newer design sends the audio into one model that produces the answer.
| Design | Path from sound to output | What the language model sees | Typical weakness |
|---|---|---|---|
| Cascade | Recognizer writes text, then an LLM works on the text | Words only, including the recognizer's mistakes | Errors from the first step carry into the second |
| Speech LLM | Audio goes into one model that responds | The audio, with or without a transcript | Needs audio-specific training and more compute |
A concrete example
Say you record: "Remind me to call Ana before the Friday review." In a cascade, the recognizer produces that sentence and the LLM can only use those words. A speech LLM receives the audio itself, so it can in principle use stress and pauses as well. Whether it does depends on how it was trained, so the difference is a design property, not a guaranteed gain.
Latency: how fast the answer starts
A speech LLM can respond quickly. Qwen2.5-Omni 7B, an end-to-end speech language model, had a 90th percentile time to first token of about 40 milliseconds on one H100 GPU 2. That figure comes from a data-center GPU in a voice assistant benchmark. A phone or laptop has far less compute, so the number says what is possible on that hardware, not what your device will do.
Why it matters for dictation
Many dictation tools are built as a recognizer followed by an optional language model, so the cascade is the common pattern. Speech LLMs matter most for voice assistants and voice agents, which must answer quickly and act on what they hear. Speech research toolkits now cover this category too. NVIDIA describes its NeMo Speech toolkit as covering automatic speech recognition, text to speech and speech language models 3.
For dictation, the practical question is what happens to your audio. A speech LLM that runs in the cloud sends the recording to a server, while an on-device model keeps it local but may be smaller. Our guide to speech to text and LLMs compares these designs in more detail.
Related terms
- Large language model (LLM): the text model that a speech LLM extends to audio.
- Automatic speech recognition (ASR): the speech-to-text task that the cascade design relies on.
- Speech-to-speech model: a model that listens and replies in speech.
- Voice agent: a system that uses speech input and output to complete tasks.
- Generative error correction (GER): a text-only LLM step that corrects recognizer output.
Guides that cover it
Evidence
Every number on this page, with where it comes from. Open a fact to see the exact words in its source.
- 1.
- 2.
- 3.
Related terms
- Speech to Text AI: Dictation, Assistants and Voice AgentsThe topic this term belongs to
- AI text detectorAn AI text detector is software that estimates whether a passage was written by a language model, usually from the words alone. Detectors make mistakes in both directions, so a score is weak evidence about how a text was made.
- Generative AIGenerative AI is the class of AI models that create new content, such as text, audio or images, from patterns in their training data. The NIST profile defines it as models that emulate the structure and characteristics of input data to generate derived synthetic content.
- Generative error correction (GER)Generative error correction (GER) passes a speech recognizer's output, often its N-best list of alternative transcripts, to a large language model, which rewrites the transcript to fix errors. It can recover words the recognizer never offered, but it can also change words that were already right.
- Intent recognitionIntent recognition is the step that decides what a spoken request is asking for, such as setting a timer or sending a message, and which details the action needs. Voice assistants and voice agents rely on it before they act.
- Large language model (LLM)A large language model (LLM) is a language model with a very large number of parameters, trained on huge amounts of text to predict and generate language. Google's glossary defines the minimum as a language model having a very high number of parameters.
- Machine learningMachine learning is a method in which a program learns its behavior from example data rather than from rules written by hand. Google's glossary defines it as a program or system that trains a model from input data.
- Prompt (AI)In AI, a prompt is the input you give a model to get a response, such as a question, an instruction or pasted text. Google's glossary defines it as any text entered as input to a large language model to condition the model to behave in a certain way.
- Speech-to-speech modelA speech-to-speech model takes audio in and produces audio out with one model, without writing the conversation out as text in between. Most voice systems instead chain speech recognition, a language model and speech synthesis.
- Text to speech (TTS)Text to speech (TTS), also called speech synthesis, is software that converts written text into spoken audio. Modern systems use neural networks, and they can run on the device or in the cloud.
- Turn detectionTurn detection is the step that decides when a speaker has finished a turn, so a voice system knows when to respond. A pause can be a hesitation rather than an ending, so the decision trades speed against cutting the speaker off.
- Voice agentA voice agent is a voice system that listens, reasons with a language model, uses tools, and speaks back across several turns to finish a task, such as checking an order and rescheduling it. Benchmarks show that agents still fail many tasks on realistic audio.
- Voice assistantA voice assistant is software that listens for a spoken request, works out what you want, and then acts or answers, such as setting a timer or turning off a light. Dictation is different: it turns your speech into text for you to edit.
Pages that use this term
- Speech to Text AI: Dictation, Assistants and Voice AgentsTopic
- Speech to Text and LLMs: What Language Models ChangeGuide
- Do LLMs make speech recognition more accurate?Question
- Is Whisper a large language model?Question
- Generative error correction (GER)Glossary
- Large language model (LLM)Glossary
- Prompt (AI)Glossary
- Speech-to-speech modelGlossary