How do you learn speech recognition?

Learn speech recognition in four steps: build the background, read a free textbook, follow a structured course with code, and then test a model yourself. Each step is free, and each one is online. The first two steps give you the vocabulary and the theory. The last two show you how systems behave on real audio, which is where most of the learning happens.

Start with Python and basic probability

The free course and the open tools in this guide use Python, so learn enough Python to run and read scripts. You also need basic probability, because a recognizer's job is to score the most likely word sequence for a recording. You do not need a degree in signal processing to start.

Read the free Jurafsky and Martin textbook for the theory

The third edition of Jurafsky and Martin's textbook is titled Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition ¹. The authors post the draft online, and they say its chapters and slides may be used in classes: "Feel free to use the draft chapters and slides in your classes" ².

Read the chapter on automatic speech recognition first, and go back to earlier chapters when a term is unfamiliar. Treat the book as a draft. Its chapters change between releases, so check the date on the version you read.

Be careful with searches for a "speech recognition book PDF." Unofficial PDF copies of textbooks may be unauthorized, and they may be out of date. The authors' own online draft is the free and official option.

Follow a structured course with working code

The Hugging Face Audio Course is free and online. Its speech recognition chapter opens by calling speech recognition, also called automatic speech recognition (ASR) or speech-to-text (STT), one of the most popular and exciting spoken language processing tasks ³. The chapter covers pretrained models, choosing a dataset, evaluation and metrics, and fine-tuning an ASR system with the Trainer API.

Stanford's CS224S is a university course. Its page describes an introduction to spoken language technology with an emphasis on dialog and conversational systems ⁴. The page describes the Spring 2025 offering. Check the course site for the current term, because assignments and lectures change from year to year.

Practice on public data, and score your results the same way

A good first dataset is LibriSpeech, a corpus of 1,000 hours of read English speech ⁵. It is a fair place to learn how models are scored, because you can compare your numbers with published results on the same kind of audio. Once you have a model running, measure its word error rate on a file whose correct transcript you wrote yourself.

Normalize the text before you score it. The Open ASR Leaderboard's normalized transcripts remove punctuation and casing, as well as disfluencies such as fillers, before word error rate is computed ⁶. If your score uses different rules, your number cannot be compared with a leaderboard, and you should say so. The glossary entry for word error rate explains the calculation.

Read and run an open model

Whisper is a good model to study, because its code and weights are public. OpenAI released Whisper's code and model weights under the MIT License ⁷. Read the inference code, then run the model on a few of your own recordings. Our guide to running Whisper locally walks through the setup.

Running a model on your own audio teaches you more than any score on a benchmark. Scores move with the audio: on the CHiME6 test set, Whisper Large V2 had a 25.5 percent word error rate ⁸, so a score on one test does not predict another. Test on the kind of audio you care about.

What to learn first depends on your goal

If you want to build an app that transcribes audio, start with the Hugging Face course and an open model, and skip the theory until you need it. If you want the theory and the math of the older statistical methods, start with Jurafsky and Martin and then the hidden Markov model entry. If you want to do research, read the papers listed in our question on important speech recognition papers.

For a quick answer on the best open model today, see our question on which Whisper model is best. For the history of the field, which explains why these methods came about, read our history guide. For the overview of every guide in this topic, see the speech recognition overview.