Preprint
Evaluating OpenAI's Whisper ASR for Punctuation Prediction and Topic Modeling
Lucas Rafael Stefanel Gris et al., arXiv (book chapter), 2023
Read the originalRead on October 16, 2026
- What it is
- A 2023 chapter that evaluates Whisper for punctuation prediction in Portuguese. Its comparison table reports F1 scores for commas, periods and question marks in earlier English work on the IWSLT 2011 benchmark, on reference text and on speech recognition output.
- How much weight it carries
- Peer-reviewed book chapter distributed as an arXiv preprint. The English figures come from earlier systems that the chapter reviews, not from the chapter's own experiments, and they are for TED talk benchmarks rather than dictation. The F1 score is per punctuation mark.
Facts taken from this source
- On speech recognition output from the IWSLT 2011 English benchmark, the best reported systems reached 66.3% F1 for commas.
“Results on ASR test set for Period were the best (82.3%); for Comma and Question it was obtained 66.3% and 63.4%, respectively.” (Results on the ASR test set (Section on English results))
- On the reference (correct) transcript of the IWSLT 2011 English benchmark, the best reported comma F1 was 76.7%.
“Results on Reference test set for Comma were the best: 76.7% F1” (Results on the reference test set (Section on English results))
- On speech recognition output from the IWSLT 2011 English benchmark, the best reported systems reached 82.3% F1 for periods.
“Results on ASR test set for Period were the best (82.3%); for Comma and Question it was obtained 66.3% and 63.4%, respectively.” (Results on the ASR test set (Section on English results))
- On the reference (correct) transcript of the IWSLT 2011 English benchmark, the best reported period F1 was 88.9%.
“Results on Reference test set for Comma were the best: 76.7% F1; for Period, F1 was 88.9% and for Question 87.8%.” (Results on the reference test set (Section on English results))
- On speech recognition output from the IWSLT 2011 English benchmark, the best reported systems reached 63.4% F1 for question marks.
“Results on ASR test set for Period were the best (82.3%); for Comma and Question it was obtained 66.3% and 63.4%, respectively.” (Results on the ASR test set (Section on English results))
- On the reference (correct) transcript of the IWSLT 2011 English benchmark, the best reported question mark F1 was 87.8%.
“Results on Reference test set for Comma were the best: 76.7% F1; for Period, F1 was 88.9% and for Question 87.8%.” (Results on the reference test set (Section on English results))