How do you calculate word error rate?

Word error rate is the sum of three error types divided by the reference word count

To calculate word error rate, compare the machine's output with a correct reference transcript, count the three kinds of error, add them up and divide by the number of words in the reference. The formula is WER = (S + D + I) / N, where S, D and I are the substitutions, deletions and insertions, and N is the reference word count. Morris and colleagues describe word error rate as the de facto standard index for recognition, defined as the proportion of word errors to words processed ¹. Koenecke and colleagues use the same denominator, the total number of words in the correct transcript ².

For the broader meaning of the score, see the guide to word error rate, the overview of speech recognition and the glossary entry.

Four steps turn two texts into a score

1. Write the reference. Use a careful transcript of the same audio. Apply the same text rules to it that you will apply to the output, such as case, punctuation and numbers.
2. Align the two texts. Pair each output word with the reference word it matches, or mark it as an error. Finding the alignment with the fewest edits is the one step that takes a computer, and it is covered below.
3. Count the three error types. A substitution is a reference word replaced by a different word. A deletion is a reference word missing from the output. An insertion is an extra word the speaker never said.
4. Divide. Add the three counts and divide by the number of reference words. Multiply by 100 if you want a percentage.

A worked example has one error of each type

The reference below has 10 words. The output is the same sentence with three changes.

This is a made-up example, chosen so that every error type appears once. In our test with the jiwer package, version 4.0.0, the same pair returns 0.3, matching the hand count. The question on calculating WER in Python shows the code.

The denominator is the reference, so the score depends on the human transcript

The divisor is the reference word count, not the output. That is why the score is fixed by the transcript a person wrote, and why a different reference gives a different score for the same output. Two consequences follow. First, a recognizer cannot lower its score by writing less, because deleting words adds deletions to the numerator. Second, the score is not a share of correct words. Accuracy is one minus WER, a different quantity. The question on WER and accuracy explains the difference.

The alignment is an edit distance, and tools can split the errors differently

Finding the three counts means finding the alignment that needs the fewest edits. That is the minimum edit distance, the same idea as Levenshtein distance, applied to words instead of letters. The jiwer package computes its measures this way ³.

The weights given to each edit affect which alignment wins when several are close. Morris and colleagues state that equal weights are optimal under their assumptions ⁴. The NIST sclite scorer uses different costs: 0 for a correct word, 3 for an insertion, 3 for a deletion and 4 for a substitution, as its version 1.2 manual gives ⁵. So two tools can report the same total from different splits between substitutions, deletions and insertions. If you publish a breakdown, name the tool.

Normalize both texts the same way before you count

Case, punctuation and number formats change the count. A recognizer that writes "100" against a reference that says "one hundred" gets an error, even though the meaning is the same. Kacprzak and Fraś report that the effect of number normalization may be substantial, more than 2 percentage points on some datasets ⁶. Pick one set of rules, apply it to the reference and to every output, and write the rules down.

Report the tool, the rules and the word count with every score

A bare percentage is hard to check. When you report a WER, give:

• the tool and its version, since the alignment weights differ between tools;
• the normalization rules for case, punctuation, numbers and filler words;
• the reference word count (N), so the reader can see how much a single error weighs.

With a 10-word reference, one error moves the score by 10 percentage points. With a 1,000-word reference, it moves the score by 0.1 percentage points. Short test clips therefore produce noisy scores. For a Python version of the same steps, see how to calculate word error rate in Python. For what a good score looks like, see what is a good word error rate.