Speech translation

Speech translation takes words spoken in one language and produces the same meaning in another language, written or spoken. Product menus may call it voice translation. When a dictation tool writes the words in a different language from the one you spoke, the act is speech-to-text translation. The input is audio, and the output is in a different language. Dictation writes what you say, but speech translation changes the language on the way out.

Most speech translation is two systems chained together

The conventional design is a cascade. A speech recognizer converts the audio into text in the source language, and a translation model converts that text into the target language. The SeamlessM4T authors describe conventional speech-to-speech systems as relying on cascaded systems that perform translation progressively ¹. Each stage works from the output of the one before, so the first stage's mistakes become the second stage's input.

The 2022 study by Tran and colleagues calls error propagation from the speech recognition system a fundamental challenge for cascaded speech translation that still remains ². Newer models try to skip the text step and map speech directly to translated text. Our guide to speech to text with translation compares the tools that use each design.

A concrete example: a Spanish sentence into English text

Suppose you say, in Spanish, a sentence about booking a table for two. A cascade first writes the Spanish words. If the recognizer hears the singular noun as a plural, the translator receives a different sentence and produces "tables" where you meant "a table". Neither stage looks wrong on its own, so you only notice the error when you read the English output. Read translated text against the original before you send it.

Translation quality drops for less common languages

Accuracy varies by language pair. On the CoVoST 2 test of speech translated into English, zero-shot Whisper scored 36.2 BLEU points for high-resource languages ³ and 25.2 BLEU points for low-resource languages ⁴. BLEU is an automatic score that compares machine output with reference translations. These numbers compare systems on one test, so they do not predict how a particular sentence you speak will come out. The low-resource figure is also an average, and single languages can fall further.

Why it matters for dictation and typing

Dictation keeps your words in the language you spoke. Speech translation puts them into another language, which makes two sources of error: the recognizer's mistakes, then the translator's. For a message or notes, a quick read is usually enough. For a contract, a medical history or a name that must be spelled exactly, dictate in the original language, check the text, and translate it separately.

Speech translation also depends on the language setting. The recognizer must know which language you are speaking, so a wrong locale can make the first step fail before translation begins.

Related terms

• Automatic speech recognition (ASR): the recognition step inside a cascade.
• Word error rate (WER): the usual measure of recognition errors, which carry into translation.
• Locale: the language and region setting the recognizer expects.
• Dialect: a regional variety whose speech is often harder to recognize and translate.

Guides and questions that cover it

• Guide: speech to text with translation on phones and computers.
• Guide: how to translate audio and video to English text.
• Question: how accurate is speech to text translation?
• Question: which APIs combine speech to text and translation?
• Overview: dictation languages, accents and voice translation.