Preprint
Are Caption Metrics Broken? Latency, Deaf and Hard of Hearing User Ratings, and Bias across Technologies
Bernard Thompson et al., arXiv (preprint), accepted for ACM ASSETS 2026, 2026
Read the originalRead on October 16, 2026
- What it is
- A 2026 U.S. online survey of 216 validated deaf and hard of hearing participants, who rated TV captions and automatic (ASR) captions of live TV clips against three caption quality metrics, including word error rate, and across caption delays.
- How much weight it carries
- Preprint accepted at a peer-reviewed conference, from a research group that works on deaf accessibility. The test material is live TV, not conversation or phone calls, and the findings describe ratings of captions rather than a product's accuracy.
Facts taken from this source
- The same 2026 survey found that caption latency, the delay between speech and text, significantly affects the viewer experience, especially the 7 to 12 second delays typical of live TV.
“caption latencies significantly impact the viewer experience, especially typical 7-12-second TV delays.” (Abstract)
- In the same 2026 survey, the three caption quality metrics, including word error rate, were moderately to highly correlated with viewer ratings of TV captions, but far less so for automatic captions.
“For TV captions, all three metrics were moderately-to-highly correlated with viewer ratings, but far less so for ASR captions, making them far from technology-neutral.” (Abstract)
- In a 2026 online survey of 216 validated deaf and hard of hearing participants, live TV captions and automatic captions synchronized with the audio were rated similarly.
“Results show that TV and ASR captions were rated similarly.” (Abstract)