Peer-reviewed research
Evaluating the Usability of Automatically Generated Captions for People who are Deaf or Hard of Hearing
Sushant Kafle and Matt Huenerfauth, ASSETS 2017, ACM SIGACCESS Conference on Computers and Accessibility, 2017
Read the originalRead on October 16, 2026
- What it is
- A study from the Rochester Institute of Technology that proposed the ACE metric for judging automatic captions and tested it with 30 deaf and hard of hearing users, who preferred captions that ACE ranked higher. The paper won the conference's best paper award.
- How much weight it carries
- Peer-reviewed and award-winning. The sample is 30 participants and one set of caption texts, and it is a 2017 study of ASR systems from that period.
Facts taken from this source
- In a 2017 study of 30 deaf and hard of hearing users, the authors' new caption metric, ACE, correlated 0.74 (Spearman rho) with their ratings of caption usability.
“The correlation coefficient was found to be rho = 0.742791 with a p-value <0.0001.” (Results, hypothesis H2a)
- In the same 2017 study, word error rate, the standard speech recognition measure, correlated only 0.11 with the same ratings.
“For 𝑟RS= 0.742791 and 𝑟RT= 0.108852, we found a significant difference between the two coefficients” (Results, hypothesis H2b)
Pages that rest on this source
- Speech to Text for Deaf and Hard of Hearing PeopleGuide
- Is automatic captioning accurate enough for deaf people?Question
- Word Error Rate (WER): How to Calculate and Read ItGuide
- Speech Recognition Benchmarks and LeaderboardsGuide
- Captions and Subtitles: Formats, Rules and ToolsGuide
- Are automatic captions good enough for accessibility rules?Question
- What is a good word error rate?Question
- What is the most accurate speech-to-text model?Question
- Word error rate (WER)Glossary