Fact · Accessibility and health
In the same 2026 survey, the three caption quality metrics, including word error rate, were moderately to highly correlated with viewer ratings of TV captions, but far less so for automatic captions.
Checked against its source on October 16, 2026
In the source's own words
“For TV captions, all three metrics were moderately-to-highly correlated with viewer ratings, but far less so for ASR captions, making them far from technology-neutral.”
Are Caption Metrics Broken? Latency, Deaf and Hard of Hearing User Ratings, and Bias across Technologies, Bernard Thompson et al., arXiv (preprint), accepted for ACM ASSETS 2026, 2026, Abstract.
Open the original- What it measures
- The three metrics were word error rate (WER), the Automated Caption Evaluation metric (ACE2) and the Number, Edition and Recognition metric (NER). Each was compared with viewers' own ratings.
- What it does not show
- The abstract does not give the correlation values for each metric. The authors conclude the metrics are not technology-neutral, which means a metric that predicts ratings for one kind of caption may not predict them for another.