How Speech Recognition Works, From Sound to Text

Speech recognition is a search problem. The system has a stretch of audio and looks for the word sequence that best explains it. It scores many candidate sentences against two questions: how well does each one match the sounds, and how plausible is it as language? The best-scoring sentence is the transcript.

Every system follows the same steps: the sound is sampled into numbers, the numbers are turned into a spectrogram, models score sounds and words, and a decoder picks the best sentence. Older systems kept those parts separate and trained each one on its own. Newer systems increasingly learn the whole mapping from audio to text in one neural network, but the same stages still explain the errors you see. This guide walks through each stage with the numbers from published studies, and ends with the errors that each stage produces. For the wider picture, start with our overview of speech recognition. For the terms used here, see the automatic speech recognition entry.

Speech recognition finds the most probable sequence of words for a stretch of sound

The classic formulation, taken from statistical speech recognition, picks the word sequence W that maximizes the probability of W given the audio. Bayes' rule splits that into two parts: the probability of the audio given the words, which is the job of the acoustic model, and the probability of the words themselves, which is the job of the language model. The decoder combines both scores and keeps the sentence with the highest total.

End-to-end neural systems do not write this product out. They learn one function from audio to text, but the question they answer is the same: which words most probably produced this sound?

Accuracy is measured with word error rate (WER): the number of substitutions, deletions and insertions needed to fix a transcript, divided by the total number of words in the correct transcript. A 2020 study of commercial systems uses that denominator ¹. A WER of 5 percent means about one word in twenty is wrong. Our guide to word error rate shows how to calculate it on your own audio.

A microphone signal becomes thousands of numbers per second, usually sampled at 16 kHz for recognition

A microphone turns air pressure into a continuous electrical signal. An analog-to-digital converter measures that signal at fixed intervals, and each measurement is one sample. The number of measurements per second is the sample rate.

For dictation, the usual rate is at least 16 kHz. Bauer and colleagues state that dictation and other large-vocabulary applications principally use a sampling rate of at least 16 kHz, especially in noisy environments ². At that rate a minute of audio is 960,000 samples ³.

The rate matters for accuracy, not just for file size. In the Aurora standardization tests, as cited by Bauer et al., recognition of noisy speech improved by a 14 percent average relative gain when the audio was sampled at 16 kHz instead of 8 kHz ⁴. Telephone-style audio is narrowband, so a call carries less of the speech signal than the same words spoken into a good microphone.

A recognizer can resample its input to the rate it was trained on. Whisper, for example, re-samples all audio to 16,000 Hz ⁵.

Feature extraction turns those numbers into a spectrogram: sound energy by frequency, in slices of about 10 milliseconds

The raw samples change too quickly, and too much, for a model to read directly. The first processing step groups them into short windows and measures how much energy the sound carries in each frequency band. Stacking those measurements over time gives a spectrogram: a picture with time along one axis, frequency along the other, and brightness for energy.

Recognizers space the windows at a fixed step. In the Whisper paper, the input is an 80-channel log-magnitude Mel spectrogram computed on 25-millisecond windows with a stride of 10 milliseconds ⁵. The Mel scale spaces the frequency bands the way human hearing does, with finer bands at low pitches where speech carries most of its information.

The 10-millisecond stride means the model sees 100 frames for every second of audio ⁶. Each frame is the unit that the acoustic model scores, which is why a model's "frame rate" is a useful number to know when you compare systems.

Before this step, many systems also run a voice activity detection check, which separates speech from silence and background noise so that the recognizer does not spend time on quiet passages.

The classic pipeline splits the job between an acoustic model, a pronunciation dictionary and a language model

The classic systems, which ran most recognition for decades, had three learned parts and one search. Each part answers a different question, and each produces its own kind of error.

The acoustic model was usually a hidden Markov model (HMM) whose states emitted the frames. Rabiner's 1989 tutorial defined the three problems that had to be solved before such a model could be used on speech: evaluating the likelihood of an observation sequence, finding the best sequence of model states, and adjusting the model's parameters to fit the signal ⁷. Later acoustic models replaced the HMM's emission probabilities with deep neural networks. In Apple's 2017 description of its "Hey Siri" wake-word detector, a deep neural network turns the acoustic pattern of the voice into probabilities over speech sounds ⁸, which is the same role the acoustic model played in the older systems.

The pronunciation dictionary links words to sounds. It is what lets a system spell out "Nguyen" or "Siobhan" once someone has listed their pronunciation. A word that is missing from the dictionary cannot be recognized, however clean the audio.

The language model is the part that makes a sentence sound right. It scores word sequences, so when two words sound the same, the more plausible one wins. Words that sound identical but differ in meaning or spelling are called homophones ⁹, and they are one reason a language model is needed. A 2007 study of foreign-accented English found that the best combination of pronunciation modelling and acoustic adaptation cut word error rate by 46 to 71 percent relative ¹⁰, which shows how much these parts matter when the speaker does not match the training data.

A decoder searches a huge space of possible sentences for the one that best fits both the sound and the language

The decoder is the search. It takes the acoustic scores frame by frame and the language model's word scores, and it looks for the path through all possible words that gives the best total. Rabiner's "determination of a best sequence of model states" is this search in its original form ⁷. In practice the number of candidate sentences is far too large to list, so the decoder keeps only the most promising paths at each step and drops the rest. That pruning is a large part of why recognition can run in real time.

Decoders often output an N-best list, a short set of the top candidate sentences, and a second model can rescore it. A 2023 benchmark found that a large language model can correct words that are missing from the recognizer's N-best list, because it generates text rather than only choosing among candidates ¹¹.

The decoder also explains a practical fact: a system can be accurate on the sounds and still produce a poor transcript if its vocabulary or language model does not cover the topic. Medical, legal and product names are the usual casualties.

Modern end-to-end neural networks learn the whole mapping from audio to text in one model

The change from the classic pipeline began with a training method. In 2006 Graves and colleagues wrote that recurrent networks needed "pre-segmented training data, and post-processing to transform their outputs into label sequences," and presented a method that trains a network to label unsegmented sequences directly ¹². That method, connectionist temporal classification (CTC), let one network learn the alignment between audio and text on its own. In 2012 Graves introduced an end-to-end transduction system for recurrent networks that removes the need for a predefined alignment between input and output ¹³.

Today's large systems follow the same idea at scale. Whisper was trained on 680,000 hours of multilingual and multitask audio with transcripts ¹⁴. Its authors chose an encoder-decoder Transformer, which reads the spectrogram and writes the text, as the architecture that "has been well validated to scale reliably" ¹⁵. There is no separate pronunciation dictionary or hand-built language model in that design. The model learns those patterns from the data.

Accuracy depends on the test. On clean, read English the best systems now make few errors. The Conformer, a network that combines convolutions with self-attention, reached word error rates of 2.1 percent on LibriSpeech test-clean and 4.3 percent on test-other without a language model, and 1.9 and 3.9 percent with one ¹⁶. The Open ASR Leaderboard paper observes that Conformer-based encoders paired with transformer-based decoders achieve the best average word error rate across its tests ¹⁷.

Those are read sentences. Accuracy falls on speech that differs from the training set, and it differs by language. The Whisper large-v2 model's error rates on the Fleurs benchmark show the spread:

The English result is 4.2 percent ¹⁸, Spanish 3.0 percent ¹⁹, French 8.3 percent ²⁰, Hindi 21.5 percent ²¹, Swahili 39.3 percent ²² and Yoruba 94.8 percent ²³. The same model, on the same benchmark, is far more accurate in some languages than others. The amount of training audio for each language is one likely reason, though these results do not measure it. For a model's family and sizes, see our guide to Whisper.

Punctuation, capitals and numbers written as digits are added by the model or by a formatting step afterwards

Many recognizers' raw output is plain words with little or no punctuation. The step that turns that into readable text is separate from the sound-to-word search. Many engines add punctuation, capitals and number formats in a post-processing step, often called inverse text normalization, which writes spoken numbers as digits and spoken abbreviations in their written form. Our glossary entry on inverse text normalization gives the details.

End-to-end models can learn punctuation directly, because their training transcripts already contain it. The output then looks like the transcripts the model was trained on, which is useful when those transcripts are carefully punctuated and a problem when they are not. The formatting step is where you can usually change how numbers and dates appear, and where some "the software wrote it wrong" complaints come from a formatting setting.

Each stage explains a familiar error: homophones, misheard names and words run together

When a transcript goes wrong, the error usually points back to one stage. This table links the common symptoms to their likely cause.

The audio itself sets the ceiling. The authors of one study note that automatic speech recognition accuracy can be affected by speech fidelity, ambient noise and microphone quality ²⁴.

Speakers matter too. In Google's Project Euphonia study of 432 people with disordered speech, personalized recognition models reached a word error rate of 4.6 percent, against 31 percent for general speaker-independent models ²⁵. The general model had not learned these voices, so its acoustic model did not match them. The same kind of mismatch can also affect accented speech.

End-to-end models have an error that the classic pipeline did not. A 2024 study found that roughly 1 percent of Whisper's audio transcriptions contained entire hallucinated phrases or sentences that did not exist in the audio ²⁶. Because the model writes text rather than choosing among candidate words, it can produce a sentence that reads well and was never spoken. Check the output of any end-to-end system against the audio when accuracy matters.

Running the model on your device or on a server changes speed, cost and privacy, not the basic method

The same stages run whether the recognizer is in a data centre or on your laptop. What changes is the size of the model, the hardware, and where the audio goes.

Size is the first constraint. In 2019 Google moved its Gboard voice typing onto the phone with an 80 MB speech recognizer that works when the phone is offline ²⁷. Its earlier server-side recognizer used a search graph of almost 2 GB, which was too large for a phone and was why it needed a connection ²⁸. Apple says that transcription with its SpeechAnalyzer API is entirely on the device, although the language models must first be downloaded ²⁹.

Streaming is the second. A system that transcribes live must commit to words before it hears the rest of the sentence, and a 2024 study found significantly lower quality for streaming recognition, the mode used for live events ³⁰. A file transcribed after the fact can use the whole recording before it commits to any word.

Privacy is the third. On-device recognition keeps the audio on the machine that recorded it. Yap, which we make, is an on-device dictation app for Mac and Windows, so its recognition runs on your computer. For how that works in practice, see our guide to on-device speech recognition and offline dictation on Mac.

Whichever way you run it, the stages in this guide are the same. A sample-rate problem, a missing word in the dictionary, a homophone in the language model and a hallucinated sentence all have the same causes on a phone, in a browser or on a server. To test a recognizer on your own voice, try the free online dictation or audio to text tools, then measure word error rate on a passage you know by heart.

Where to go next

Start with the question on whether speech recognition is AI if the terms are new, or the question on whether speech recognition is part of NLP if you are reading about language models. The question on what speech recognition is used for covers the real-world applications. For the models themselves, read the guides on speech recognition models, how accurate speech recognition is and the history of speech recognition.