Real-Time Speech Recognition: Streaming and Latency

Real-time speech recognition turns speech into text while the speaker is still talking. A streaming recognizer receives audio in small pieces, returns tentative words while the speaker is still talking, and revises them as more audio arrives. Batch recognition takes a finished recording and returns one transcript at the end, usually with fewer errors. That trade-off is the basis of everything else in this guide: a streaming model can see only a little of the audio that follows each word, so it is less accurate than a model that hears the whole recording ¹.

For live use, three numbers decide whether a system feels instant. The first is how soon the first words appear. The second is how soon the final text is ready after you stop. The third is how long the system waits before it decides you have finished. Vendors measure these from different start points, so this guide defines each one before it compares any figures. For the wider field, read our overview of speech recognition, and for error scoring see word error rate.

Streaming recognition shows words while you speak, while batch recognition waits for the whole recording

A streaming session stays open while you talk. The client sends audio as it is captured and receives results as they become available. Google Cloud says its streaming recognition lets you stream audio and receive results "in real time as the audio is processed" ².

Microsoft's Azure Speech service offers a real-time mode and a batch mode, among others. Real-time transcription gives "Instant transcription with intermediate results for live audio inputs" ³. Batch transcription handles prerecorded files. You submit the files, then collect the results later. At peak hours a batch job can take up to 24 hours to finish ⁴.

Batch is not a faster version of streaming. Streaming audio must arrive at roughly the speed people speak, because Google requires that "audio must be sent at a rate that approximates real time" ⁵. Google's limits show the difference in what each mode can take in one request:

Batch jobs also have a queue. Microsoft says that at peak hours a batch job "might take up to 30 minutes" to start and up to 24 hours to finish ⁴. If your audio is already recorded and you can wait, batch is usually the more accurate choice. If someone is waiting on the words, you need streaming.

Streaming models see only a little future audio, so they give up some accuracy for speed

A recognizer decides what a word is partly from the sounds that come after it. A batch system can use the whole recording before it commits to any word. A streaming system must commit to words before the rest of the sentence has been heard. That is why streaming is less accurate in general. A 2024 study comparing commercial services found significantly lower quality for streaming speech recognition, the kind used for live events ¹.

The same trade-off shows up when a system decides a sentence is over. In a 2020 Google paper on an on-device recognizer, the authors used an acoustic endpointer, a separate detector that closes the microphone. With the endpointer alone, the word error rate rose from 6.2 to 7.4 percent in order to reach a median endpointer latency of 450 ms ⁸. The authors accepted that higher error rate for the faster close, so the speed is a trade-off, not a free gain.

Model design decides how much of the trade-off you pay. A transducer model, described in the glossary entry on RNN transducers, was built to emit words as audio arrives. In a 2019 study of streaming recognition for mobile devices, the authors report that their RNN transducer "can outperform a conventional CTC-based model in terms of both latency and accuracy" in a number of test categories ⁹. The architectures are explained in how speech recognition models work.

Latency is several numbers: time to first partial, time to final text and end of speech to text

Each vendor reports a different number, so compare only those that measure the same thing. The table below sets out the four that appear most often.

The first partial matters most for live captions. End-of-turn latency matters most for a voice agent, because it decides when the agent can answer. For dictation, the wait after you stop is closest to endpointer latency plus the time for the final text to arrive.

One published measurement of time to first partial is an independent benchmark from Artificial Analysis, a 2026 streaming test that measures latency from the end of speech as a voice activity detector reports it. In that test, Deepgram Flux had the fastest first partial at 0.019 seconds ¹⁴. ElevenLabs Scribe v2 Realtime took 0.13 seconds on its first partial, with a 3.65 percent word error rate ¹⁵, and AssemblyAI U3 Realtime Pro took 0.47 seconds ¹⁶.

The start point is the important caveat. Artificial Analysis starts its clock when a voice activity detector reports the end of speech, so the time that detector takes to decide is not counted ¹⁰. A slow detector would make a service look faster in this test than it feels to a user. For the question of which API is fastest overall, see the question on the fastest speech to text API and the guide to speech to text APIs compared. Deepgram's own documentation says its Nova-3 model delivers sub-300 ms streaming latency ¹⁷. That is a vendor claim, not an independent measurement.

Human tolerance sets the target. The ITU-T's G.114 recommendation says that if the mouth-to-ear delay in speech can be kept below 150 ms, "most applications, both speech and non-speech, will experience essentially transparent interactivity" ¹⁸. That is a standard for phone calls, not for transcripts, but it sets the scale of delay people notice in conversation.

Real-time factor measures throughput, not how fast words appear

The real-time factor (RTF) is the ratio of processing time to audio time. Artificial Analysis calls its version a speed factor and defines it as audio duration divided by API response time. It says that "a value above 1 means the service transcribes faster than real time" ¹⁹.

The Open ASR Leaderboard reports the inverse, RTFx, where a larger number is faster. On its English short-form test, Parakeet TDT 0.6B v2 had an RTFx of 3,390 and Whisper Large v3 had 146, both in batched GPU runs on the leaderboard's own hardware ^{20 21}.

These numbers measure throughput on batched audio, not how soon the first word appears on screen. The leaderboard paper says that token-and-duration transducer decoders "offer superior RTFx, making them better suited for long-form and batched processing" ²². A system with a high throughput can still wait a long time for its first partial word, and a system that starts quickly can still process a file slowly. Treat RTF as a capacity number for servers and batch jobs, and treat the first-partial and end-of-turn numbers as the measures for live use.

Partial results get rewritten as more audio arrives, which is why live captions flicker

A partial result is a guess. Google's v1p1beta1 API reference says that interim results are "tentative hypotheses" that may be returned as they become available, and it marks them with a flag, so the client can tell them from final text ²³. A word that appears on screen as partial can change to a different word a moment later, once the words after it arrive and the model reconsiders. That revision is the flicker people notice in live captions.

The benchmark makes the same split. It reports time to the first partial and time to the first final transcript as separate numbers ¹¹. The first partial can be quick because it is allowed to change. It can also be less accurate: OpenAI's hosted GPT Realtime Whisper model improved from a 7.5 percent word error rate on the first partial to 5.1 percent on the final transcript ²⁴.

For dictation, the practical rule is to insert only final text into a document and show partial text only as feedback. Inserting partials can leave wrong words in the text that the model later corrects, and the cleanup is more work than waiting for the final result. Techniques for clean dictated text are in the guide on how to dictate.

Voice activity detection and endpointing decide when a phrase is finished

Two components decide when a streaming system should act. Voice activity detection (VAD) labels each piece of audio as speech or silence, and a VAD model is light enough to run on every chunk of audio.

Endpointing is the decision that a turn has ended. It is the harder problem, because a pause in the middle of a sentence looks the same as a pause at its end.

Platform APIs expose the same decision as settings. Android's recognizer intent has a silence setting for when input counts as complete. The complete-silence length is "the amount of time that it should take after the recognizer stops hearing speech to consider the input complete hence end the recognition session" ²⁵.

Cutting off is a real cost. In a 2023 study of speech recognition for people who stutter, the baseline endpointer cut off an average of 23.8 percent of utterances ²⁶.

For the mechanics in more depth, see the question on how speech recognition knows when you stop talking.

Dictation needs fast final text after you stop, not word-by-word captions

Dictation and live captions have different targets. A caption reader wants words to appear quickly and keep appearing. A dictator wants correct final text soon after stopping, and does not mind if the words on screen lag while speaking. That is why a dictation app can afford a more accurate final pass than a caption system.

The timeouts that end a dictation session are a separate setting from endpointing. On iPhone, Dictation stops automatically after 30 seconds without speech ²⁷. That timeout tells you how long a dictation session stays open when you pause, not how fast the final text arrives.

For a dictated sentence, the wait after you stop is the sum of the endpointing decision and the final result. Measure it on your own microphone and in your own room, because a noisy room makes the endpointer hesitate. The guides on dictation technique and live transcription on phones cover the practical side.

Each network round trip adds latency that on-device recognition avoids

A cloud recognizer adds at least one trip across the network. Audio goes up, text comes back, and each leg adds delay that depends on distance, congestion and the server's queue. Latency research often separates the network from the recognizer. Sainath et al. define endpointer latency as the time for the microphone to close after a user stops speaking ¹³. That definition covers the endpointer alone, so it says nothing about the trip to a server. The comparison therefore rests on the recognizer's own timing, measured on their test set.

The phone-call standard shows how little room there is. G.114 puts the limit for mouth-to-ear speech delay, the time from one speaker's mouth to the other's ear, below 150 ms ¹⁸, and a network trip spends part of that budget. A speech recognizer that is far away on the network starts with a deficit before it has transcribed a single word. On-device recognition removes that trip, which is why many dictation tools run locally when the hardware allows. The trade-off is model size: a model that fits on a phone or laptop is smaller than the largest cloud models, so the choice is between speed and the best possible accuracy. The guide on on-device speech recognition sets out the trade-off in more detail.

Cloud streaming still has advantages. A server can run a larger model, update it without an app release, and handle long sessions. Google's streaming sessions are limited to about 5 minutes ⁶, so a long meeting needs a service that reconnects, and that reconnection is a latency event of its own.

Transducers and CTC models stream natively, while Whisper-style models stream only by chunking

Two families of model suit streaming. Connectionist temporal classification (CTC) was designed to label an unsegmented input sequence directly. The CTC entry in the glossary gives the basics. RNN transducers extend the idea so that the model emits symbols as audio arrives. Graves wrote in 2012 that traditional recurrent networks "require a pre-defined alignment between the input and output sequences", and introduced a transduction system that removes that need ²⁸. Those designs output text frame by frame, which is why they can produce partial words quickly.

Whisper is a different design. Its paper says its models are trained on 30-second audio chunks and cannot take longer audio as a single input ²⁹. A Whisper-based streaming app therefore re-runs the model on a moving window of audio. The whisper.cpp streaming example does this by sampling audio "every half a second" and running transcription continuously ³⁰. Chunked decoding is a workaround, and the window size sets a floor on delay: the model cannot return the end of a sentence until it has the audio for it.

OpenAI's hosted realtime transcription models handle turns differently. OpenAI says that "gpt-live-transcribe and gpt-realtime-whisper require turn detection to be omitted or set to null" ³¹, so the client must decide when a turn ends. The open-source Whisper code has a different check, the no speech threshold, which decides whether a decoded window is silence. Its docstring says the segment is treated as silent when the no speech probability is "higher than this value" and the average log probability is below the threshold ³². That check runs on a window the model has already decoded, so it does not tell a live app when the speaker has finished.

Whether Whisper can keep up with live speech depends on the model size and the hardware. The question on whether Whisper can transcribe in real time works through the options, and the guide on running Whisper locally covers the setup. For the model families themselves, see speech recognition models and the history of speech recognition.

What to measure before you choose a real-time setup

Pick the number that matches your use. For live captions, measure the time to the first partial and the lag of the transcript. For a voice agent, measure end-of-turn latency, because it decides how quickly the agent can respond. For dictation, measure the wait from the moment you stop to the final text, and count the errors in that text, because a fast wrong result costs more than a slightly slower right one.

Use your own audio. Benchmarks use their own recordings, and a vendor's sub-300 ms claim ¹⁷ does not describe your room or your accent. Score accuracy with word error rate on your own sentences, and read the guide to speech recognition benchmarks before you rely on any leaderboard. The question of how accurate speech recognition is covers the scoring in more detail.