How does speech recognition know when you stop talking?

Speech recognition does not know you have stopped in any deep sense. It watches the audio for silence. A voice activity detector labels each short piece of sound as speech or silence, and an endpointer waits for silence to last long enough before it closes your turn and returns the final text. The length of that wait is the setting that matters most, because it trades a fast finish against cutting you off mid-sentence.

A voice activity detector labels each slice of audio

The first step is voice activity detection (VAD). A VAD model is small enough to run on every slice of audio, and it outputs a yes or no for speech. OpenAI describes its server-side version in these terms: it "uses periods of silence to automatically chunk the audio" ¹. The detector cuts the stream at the silences, and each cut becomes a candidate end of turn.

The detector also needs a threshold, which sets how loud a sound must be to count as speech. OpenAI says that "a higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments" ². Background noise that the detector reads as speech can keep it from seeing silence, which delays the end of your turn.

The silence timer decides when a turn ends

The endpointer is the decision step. It asks how long the silence has lasted, and the answer is a timer. OpenAI's Realtime guide describes its silence setting, the duration of silence that marks the end of speech, and says that shorter values mean turns are detected more quickly ³. Shorter is faster, but a short timer also splits a sentence at a thinking pause.

Google's streaming API sets a floor on this timer. Its voice activity timeouts "must be greater than 500ms and less than 60s" ⁴, so you cannot set a timer shorter than half a second on that service.

Faster endpointing has a measured cost. In a 2020 Google study of an on-device recognizer, the acoustic endpointer alone had a median latency of 450 milliseconds, and the word error rate rose from 6.2 to 7.4 percent ⁵. The authors chose that setting after testing the options, so it is a balance, not a free gain. For the wider picture of that wait, see the real-time guide.

The cost can be high for some speakers. In a 2023 study of people who stutter, the baseline endpointer cut off an average of 23.8 percent of utterances ⁶. A timer that suits fluent speakers can split the words of someone who pauses mid-phrase.

Whisper's no-speech threshold skips silent windows, not turns

In the open-source Whisper code, the setting is called no_speech_threshold. Its command-line option, --no_speech_threshold, defaults to 0.6 ⁷, and it is a probability. Whisper decodes audio in 30-second windows ⁸, and the check compares the no-speech probability against that value.

The check is applied to a window that the model has already decoded. Whisper's transcribe code treats a segment as silence, and skips it, only when its no-speech probability is above the threshold and its average log probability is below a second threshold ⁹. The skip removes silent windows from the transcript, which is why it is useful for long recordings, and why it does not help a live app. The app still has to decide that you have finished speaking before it can show the final text.

In plain terms, the no-speech threshold is a filter for silence in a window that has already been decoded. It does not tell a microphone when to close. For that job a live system uses a separate VAD and endpointer, as described above. The question on whether Whisper can transcribe in real time covers how these pieces fit together in a Whisper setup.

What to do when the end of your turn is wrong

If your text is cut off, the timer is too short for how you speak. If the final text arrives late, the timer is too long or the room is noisy enough to keep the detector from seeing silence. Adjust one setting at a time and test each change on the same clips.

1. Find the setting. Look for a silence timeout, an end-of-speech timeout or a voice activity threshold in the tool or API you use.
2. Record your own pauses. Read a few sentences with the pauses you normally take, including one mid-sentence.
3. Count cut-offs and delay. Note how often a sentence is split and how long the final text takes to appear. Both numbers matter.
4. Raise the timer if you split sentences. Lower it only if the final text is too slow and the cut-offs are rare.
5. Consider push to talk for hard cases. A key that you hold while you speak removes the guessing about when you have finished.

For the speed of the final text, read the question on the fastest speech-to-text API. For the wider picture of live recognition, see real-time speech recognition.

For the wider field, read our overview of speech recognition.