How do you calculate word error rate in Python?

One call to jiwer.wer() returns the score as a fraction

The quickest way to calculate word error rate in Python is the jiwer package. It computes its measures from the minimum edit distance between the reference and the hypothesis ¹. Install it with pip, the route the jiwer README gives beside uv ²:

pip install jiwer

Then pass it the reference transcript and the recognizer's output:

from jiwer import wer

reference = "please book a table for two people at seven tonight"
hypothesis = "please book a cable for two at seven tonight now"

print(wer(reference, hypothesis))  # 0.3

The function returns a fraction, so 0.3 means 30 percent. In our test with jiwer 4.0.0 on Python 3.12, this printed 0.3, which matches the hand count of one substitution, one deletion and one insertion over ten reference words. The question on how to calculate WER explains each step.

process_words() shows the three error counts

A single score hides the split between error types. The process_words function returns the counts as well:

from jiwer import process_words

measures = process_words(reference, hypothesis)
print(measures.substitutions, measures.deletions, measures.insertions, measures.hits)

For the same pair this prints 1 1 1 8. The eight hits are the correct words, and with one substitution and one deletion they account for all 10 reference words. Keep the three counts when you report a result, since two systems with the same WER can make different kinds of errors.

Normalize case and punctuation before you score

jiwer compares the strings as they are. It does not lowercase them or strip punctuation by default. In our test, "Hello world." scored 1.0 against "hello world", because both words differed: "Hello" against "hello" and "world." against "world". That is two errors over two reference words. Apply the same transform to both texts:

import jiwer

normalize = jiwer.Compose( [
    jiwer.ToLowerCase(),
    jiwer.RemovePunctuation(),
    jiwer.RemoveMultipleSpaces(),
    jiwer.Strip(),
    jiwer.ReduceToListOfListOfWords(),
])

score = jiwer.wer(
    "Hello world.",
    "hello world",
    reference_transform=normalize,
    hypothesis_transform=normalize,
)
print(score)  # 0.0

The transform does not map numbers. In our test, "one hundred" against "100" scored 1.0, because the two words are different strings. If your references spell numbers out, convert the output's digits to words, or the score will count those as errors. Kacprzak and Fraś report that this kind of mismatch may be substantial, more than 2 percentage points on some datasets ³.

Write the same calculation by hand

The arithmetic is small enough to write yourself. This version fills the edit-distance table one row at a time, which keeps memory use low:

def word_error_rate(reference, hypothesis):
    ref = reference.split()
    hyp = hypothesis.split()
    # previous row of the edit-distance table, starting with insertions only
    prev = list(range(len(hyp) + 1))
    for i in range(1, len(ref) + 1):
        curr = [i] + [0] * len(hyp)
        for j in range(1, len(hyp) + 1):
            cost = 0 if ref [i - 1] == hyp [j - 1] else 1
            curr [j] = min(prev [j] + 1, curr [j - 1] + 1, prev [j - 1] + cost)
        prev = curr
    return prev [len(hyp)] / len(ref)

It returns 0.3 for the example above, matching jiwer. It has no text normalization, so apply your rules to the strings before you call it. It also raises ZeroDivisionError on an empty reference, which the next section explains.

Score many pairs as one corpus

When you have many clips, pass lists of strings. jiwer then pools the errors over all pairs and divides by the total reference words, rather than averaging the per-clip scores. In our test, two pairs with 2 and 4 reference words and 2 errors in total gave 2 / 6 = 0.333, while the average of the two clip scores gave 0.375. Long clips therefore carry more weight in the pooled figure. Say which one you report.

Empty references need a rule of their own

An empty reference divides by zero, and the jiwer README says that this case is undefined behaviour in the WER formula ⁴. Since version 4.0, jiwer defines it: when the hypothesis is non-empty, every word counts as an insertion ⁵. In our test, jiwer.wer("", "silence") returned 1, and jiwer.wer("", "peaceful silence") returned 2. That is the right result for a silent clip where the recognizer invented text, but it is a score of 200 percent, which many readers will not expect. The question on WER above 100 percent covers the reasoning.

What to do next

1. Install jiwer in a virtual environment, and record its version with every score.
2. Normalize both strings with the same transform, and write the transform down.
3. Print the substitution, deletion and insertion counts, not only the total.
4. Pool clips with a list of pairs, and state whether you report the pooled or the per-clip mean.

For more on the tools themselves, see speech recognition in Python: libraries compared, part of our overview of speech recognition. For the arithmetic behind the number, see how to calculate word error rate.