RNN transducer (RNN-T)

An RNN transducer (RNN-T) is a neural speech model that writes text one symbol at a time while it reads the audio. Graves introduced the transducer in 2012 as an end-to-end system, and noted that recurrent networks had traditionally required a pre-defined alignment between the input and the output ¹. The transducer removes that requirement, which is why it can be trained directly on audio and text.

Three networks work together

An RNN-T has three parts:

• An encoder reads the audio frames in order and builds a representation of the sound so far.
• A prediction network reads the symbols the model has already written, so it knows the words before the current one.
• A joint network combines the two and scores the next output: a character, a word piece, or a blank that means "read more audio".

Because the prediction network looks at earlier output, each choice has context that a CTC model lacks. The model can also emit several symbols from one stretch of audio before it moves on to the next.

Streaming is why it suits dictation

Google described its on-device recognizer, an RNN-T model, as one that "continuously processes input samples and streams output symbols", and said that property is welcome for speech dictation ². In a 2019 study, He and colleagues reported that their RNN transducer could outperform a conventional CTC-based model in both latency and accuracy across a number of evaluation categories ³.

Latency is the delay between saying a word and seeing it. A model that waits for the whole sentence before it writes anything feels slow when you dictate. A transducer keeps that delay short, because each word is written as soon as the model has heard enough to decide it.

The trade-off is that a streaming model sees less of the future audio when it writes each word. A file transcribed offline can use a larger model that reads the whole recording first, which usually gives a lower error rate at the cost of waiting.

TDT: a transducer that skips frames

A token-and-duration transducer (TDT) is a variant that predicts both the symbol and how many audio frames it spans, so the decoder can jump ahead. The Open ASR Leaderboard paper says TDT decoders offer superior speed, measured as RTFx, which makes them better suited for long-form and batched processing ⁴. Speed matters for transcribing hours of audio, where the real-time factor decides how long the job takes.

Related terms

• Connectionist temporal classification (CTC): the simpler training method that transducers extend.
• Conformer: an encoder design often paired with transducer decoders.
• Automatic speech recognition (ASR): the broad field this model belongs to.
• Real-time factor (RTF): the measure of how fast a model processes audio.

Guides that cover it

• Real-time speech recognition: streaming and latency
• Speech recognition models: how neural ASR works