History of Speech Recognition: From Audrey to Whisper

This history starts in 1952, when Bell Laboratories built a machine that recognized isolated spoken digits from one speaker ¹. Over the seven decades since, the field moved through statistical models, deep neural networks and end-to-end networks, and reached Whisper, a model that supports 98 languages ². No single person invented the technology. Our answer to who invented speech recognition names the teams behind each step.

Speech recognition took seven decades to go from ten spoken digits to open dictation in dozens of languages

Each milestone raised accuracy for a specific reason, and the milestones are easy to date from primary sources. The table lists the ones this guide covers, with the number that shows what changed.

Read the table with one caution: the studies use different test sets and measure different things, so the numbers show the direction of progress more reliably than the exact size of each step. The first machines were tiny in scope. The later ones were large in data and computing power, and the most recent ones are open enough that you can run them yourself. Our overview of how speech recognition works explains the parts that stayed the same across all of these steps.

1952: Bell Labs' Audrey recognized spoken digits from a single speaker

Bell Laboratories was the research arm of the telephone company, and Audrey was a digit recognizer, not a dictation system. Ninety percent sounds high, but it applied to ten digits from one voice, so it says little about the next speaker. Bell Laboratories built it in 1952 ¹⁴, and it recognized the digits zero to nine with 90 percent accuracy, but only when its inventor spoke ¹⁵.

That single-speaker limit explains much of the early history. A system that works for one voice does not transfer to the next speaker, so each new user needed a new setup. The same need came back in IBM's Tangora, a research system that had to be trained by each individual user ¹⁶.

1961 and 1962: IBM's Shoebox understood 16 spoken words

IBM's Shoebox answered the next question: could a machine recognize whole words, not just digits? IBM says engineer William C. Dersch introduced the experimental machine at a press conference in 1961 ¹⁷. It recognized the digits zero through nine and six command words, such as plus, minus, total and subtotal ³. Counting the ten digits and six commands gives the 16 spoken words that IBM's archive describes ⁴.

Shoebox's small size came from its circuit design. IBM says the machine needed only 31 transistors ¹⁸, while earlier speech recognition machines required as many as 200 transistors per word ¹⁹. IBM also says Shoebox appeared on the Today show and at the 1962 Seattle World's Fair ²⁰. The press conference and the public appearances fall in different years, so a date of 1961 or 1962 depends on which event a source counts.

A 1971 research program set a 1,000-word goal, and Harpy reached 1,011 words

The next big step was a goal, not a product. A 1971 group of scientists recommended a five-year program, and its target was a system that accepts connected speech from many speakers on a 1000-word task vocabulary ⁵. The program was a DARPA speech understanding effort that ended in 1976 ²¹.

Carnegie Mellon's Harpy came closest to the goal. The program's retrospective reports that Harpy recognized speech from male and female speakers on a 1,011-word document retrieval task, with a 5 percent semantic error ⁶. Semantic error counts mistakes in the meaning of the result, so it is not the same as the word error rate used today. Two other systems from the program, Hearsay-II and BBN's HWIM, did not meet the program's performance goal when it ended ²¹. The glossary entry for word error rate explains why that measure is the one to compare.

From the 1970s to the 2000s, statistics and hidden Markov models replaced hand-written rules

The big change was a shift from rules written by people to models that learn from data. Juang and Rabiner's 2005 history says that most practical systems were based on the statistical framework developed in the 1980s ²². The most notable tool of that decade was the hidden Markov model, which handles the timing of speech as a sequence of states.

Rabiner's 1989 tutorial shows why the model was useful. It lays out three problems an HMM must solve before it can be applied to speech ²³. Once those problems had solutions, the model could score how well a sequence of words explains a recording.

Progress was real, but the error gap stayed wide. In a 1997 review, an HMM recognizer had a word error rate of 43 percent on spontaneous telephone conversations from the Switchboard corpus ²⁴. Human transcribers had 4 percent on the same kind of speech ²⁵. Even on quiet, read speech, the same review found machine error rates often more than ten times higher than human error rates ²⁶.

In the 1990s, Dragon brought dictation to personal computers

Before 1997, dictation products either had small vocabularies or made users pause between every word ²⁷. Dragon Systems changed that in 1997, when it unveiled NaturallySpeaking, which was the first general-purpose, large-vocabulary continuous speech recognition product ⁷. At launch it cost $695 ²⁸.

Our guide to Dragon NaturallySpeaking covers the product line's later versions. For the wider story of typing by voice, see the history of dictation.

Around 2010 to 2012, deep neural networks cut error rates by large margins

The turn to neural networks brought a clear drop in error rates, as the numbers below show. In 2012, four research groups described how deep neural networks, which have many hidden layers, outperformed the Gaussian mixture models that had scored speech ²⁹. Before that change, most systems used hidden Markov models for timing and Gaussian mixtures to score each state against a short window of sound ³⁰.

The network behind this result was a seven-hidden-layer network with 2,048 units in each layer replaced the Gaussian mixture model in the acoustic model ³¹. The numbers are the clearest evidence. In the same review, a neural network reduced the word error rate of a GMM-HMM baseline from 27.4 to 18.5 percent on a Switchboard telephone task ⁸. It is one of several benchmarks in the paper, so read it as one data point rather than a result for every task. The glossary entry for the acoustic model explains which part of the system the network replaced.

Recurrent networks followed. A 2013 deep recurrent network reached a 17.7 percent test set error on the TIMIT phonetic benchmark ³². The papers behind these steps are the core of our question on important speech recognition papers.

End-to-end models merged the separate parts into one network by 2015 and 2019

Early neural networks still needed a pre-segmented training set, so the audio had to be aligned to the text before training. Graves and colleagues introduced connectionist temporal classification in 2006 to train on unsegmented sequences ³³. In 2012, Graves described a transducer for sequences and noted that recurrent networks had traditionally required a predefined alignment between input and output ³⁴.

In 2015, the Listen, Attend and Spell model produced character sequences with no independence assumption between characters ³⁵. The step that mattered for phones came in 2019. Google described a recognizer built on an RNN transducer that was compact enough to run on the phone ³⁶, and it said the on-device design matched the accuracy of its server models ³⁷. The on-device recognizer was an 80 MB model that works offline ¹¹. Google says the model continuously processes input samples and streams output symbols, a property that is welcome for speech dictation ³⁸.

In 2017, Microsoft and IBM reported human-level error rates on telephone conversations

Telephone conversation was the hardest test for a long time. In 2017, Microsoft reported that professional transcribers had a 5.9 percent word error rate on the Switchboard portion of its test data ³⁹. Its automated system reached 5.8 percent on the same portion ⁹. IBM's own recognizer reached 5.5 percent on the Switchboard subset of the Hub5 2000 evaluation, against 5.1 percent for the best human transcriber ^{10 40}.

Microsoft reported its professional transcribers and its system on the same Switchboard portion ^{39 9}, so that comparison is direct, though it covers one benchmark. The gap was not closed on every test. On the harder CallHome conversations, IBM's machine reached 10.3 percent against 6.8 percent for its best human transcriber ^{41 42}, a gap of 3.5 percentage points ⁴³. Comparing the 1997 HMM recognizer's 43 percent ²⁴ with Microsoft's 5.8 percent ⁹ compares different systems on related tests, so it shows the direction of progress, not a controlled gain. In these papers, "human parity" refers to one benchmark, not to all speech. For how these numbers should be read, see the guide to speech recognition accuracy.

Since 2022, large weakly supervised models like Whisper made strong recognition free to download

Whisper changed who could use strong recognition. Its authors trained it on 680,000 hours of multilingual and multitask audio with transcripts ¹². The OpenAI README says Whisper's code and model weights are released under the MIT License ¹³, so anyone can run it, including for commercial use. You can see how that works in our guide to running Whisper locally.

The model is trained on 30-second chunks, and the authors say it cannot take longer audio as a single input ⁴⁴, so longer recordings are handled in pieces. The results are strong on clean speech. Whisper Large V2 reached a 2.7 percent word error rate on LibriSpeech Clean ⁴⁵, and the Whisper paper puts the best LibriSpeech test-clean result at 5.3 percent in 2015 and 1.4 percent by 2021 ⁴⁶. The authors call English recognition very close to human-level accuracy, but not perfect ⁴⁷. On the harder CHiME6 test, the same model had a word error rate of 25.5 percent ⁴⁸.

Whisper also has a failure to watch for. The authors list complete hallucination among its known failure modes, where the model outputs a transcript unrelated to the audio ⁴⁹. For a reader, the practical lesson is that a model's score depends on the audio. Clean read speech produces the low numbers and harder audio produces higher ones, so test a model on your own recordings before you trust it. Check any transcript of hard audio, and see our guide to Whisper AI for what the model does well and badly.

Ten milestones mark the path from Audrey to today

1. 1952: Bell Labs builds a digit recognizer for a single speaker ¹.
2. 1961: IBM introduces Shoebox, which knows 16 words ⁴.
3. 1971: A five-year program sets a 1000-word goal for connected speech ⁵.
4. 1980s: Statistical methods and hidden Markov models become the mainstream approach ²².
5. 1997: Dragon NaturallySpeaking brings continuous dictation to personal computers ⁷.
6. 2012: Deep neural networks reduce Switchboard word error from 27.4 to 18.5 percent ⁸.
7. 2015: Listen, Attend and Spell produces character sequences with no independence assumption between characters ³⁵.
8. 2017: Microsoft and IBM report near-human results on Switchboard ⁹.
9. 2019: An 80 MB on-device recognizer works offline on phones ¹¹.
10. 2022: Whisper is released with code and weights under the MIT License ¹³.

Most of these steps share one thread: models that learn the mapping from sound to text from data, rather than from rules written by people. The glossary entry for word error rate and the guide to speech recognition benchmarks explain how to compare future results.

Yap, which we make, is an on-device dictation app for Mac and Windows. You can try browser-based transcription without installing anything with our free online dictation, and read about our download options if you want the app on your own computer. To go deeper into the field, see our question on how to learn speech recognition, and return to the speech recognition overview for every guide in this topic.