Can an ESP32 or microcontroller do speech recognition?

Yes, within limits. An ESP32 can listen for a wake word, such as a phrase that starts a device, and it can recognize a fixed list of short commands, such as "turn on the light." It cannot transcribe open speech into text the way a phone or laptop does. The limit comes from memory and compute, not from the idea of speech recognition itself.

The guide to on-device and cloud speech recognition covers the same trade-off for phones and computers. This page covers the smallest case, a microcontroller, and what each ESP32 chip can run.

An ESP32 handles a fixed command list, not open dictation

Espressif, the company that makes the ESP32, publishes a speech command model called MultiNet. It supports up to 300 Chinese or English speech commands ¹. That is the right shape for a device: a short phrase set that you define in advance, such as the rooms in a house or a timer.

The chip's memory is the main constraint. The ESP32-S3 is a dual-core processor that runs at 240 MHz, with 512 KB of internal SRAM ². Espressif also says its vector instructions speed up neural network work ³, so the S3 has hardware support that helps this kind of model. Hello Edge, a 2017 study of keyword spotting, describes the same constraint: such always-listening tasks typically run on tiny microcontrollers with limited memory and compute capability ⁴.

Open dictation does not fit. OpenAI's Whisper large model has 1,550 million parameters and needs about 10 GB of memory ⁵. Against the 512 KB of SRAM on an ESP32-S3, that is roughly 19,500 times as much ⁶. External memory helps a board, but not by enough to close a gap that size. For open speech, a phone, a laptop or a small computer such as a Raspberry Pi is the realistic choice.

The chip you pick decides which features you get

Not every ESP32 runs every model, so check the chip before you design around a feature. Espressif's esp-sr README lists the supported chips for each model. Its speech command table lists the ESP32-S3, the ESP32-P4 and the ESP32-S31 as the targets for MultiNet ⁷. The original ESP32 is not in that table.

Wake words are a different case. Espressif's esp-sr README says its WakeNet10s wake word model can run on chips without SIMD instructions, such as the original ESP32 ⁸. The same README says WakeNet9s can run on chips that have no PSRAM and no SIMD support ⁹. So a plain ESP32 can detect a wake word, but do not assume it can also run the command list.

Board designs show the range. Home Assistant's Voice Preview Edition uses an ESP32-S3 with 16 MB of flash and 8 MB of octal PSRAM, plus a separate audio chip ¹⁰. By default it can listen for three built-in wake words ¹¹. That is a wake word and a short command path on an S3 board, not open dictation.

What works in practice: wake word, then command list, then a server

A common design splits the work. The microcontroller listens for the wake word locally, so the device stays quiet until you call it. After the wake word, the device either matches a short command on the chip or sends the audio to a larger machine for anything else. This keeps the always-on part small and private, and it keeps the expensive part off the chip.

Accuracy can be affected by the speech, the background noise and the microphone quality, among other variables ¹². A poor microphone can make a microcontroller miss commands that a phone would recognize. Test with the actual microphone, the actual distance and the people who will use the device.

What to do

1. Write down the command list first. Count the phrases. If you need more than 300, the model's limit ¹, MultiNet is the wrong fit.
2. Pick the chip from the model. For command recognition, choose an ESP32-S3, ESP32-P4 or ESP32-S31 class part, and check the esp-sr documentation for the exact model your SDK version supports.
3. Confirm the memory budget for your model and your audio buffers, and check it against the chip's memory, including any external PSRAM your board has.
4. Test with real speech in the room where the device will sit. Record the miss rate for each command, not only whether the demo worked once.
5. Send anything outside the list to a bigger machine. For free-form speech, a Raspberry Pi or a computer running a model is the realistic next step. Our question on speech recognition on a Raspberry Pi covers the timings.

Related pages

• The speech recognition overview covers how the technology works and how it is measured.
• The guide to on-device and cloud speech recognition explains where the model runs and what that costs.
• What Home Assistant's Voice Preview Edition is describes the ESP32-S3 board in more detail.
• Is on-device speech recognition as accurate as cloud? covers the accuracy trade-off behind the device-versus-server choice.
• For the terms used here, see the entries on wake word and on-device speech recognition.