{"dataset":{"name":"On-device speech recognition word error rates on FLEURS","title":"On-Device Speech Recognition Benchmark on a Mac","question":"How accurate are the speech recognition engines that run entirely on a Mac, in each of 25 European languages?","url":"https://yapapp.cc/data/on-device-asr-benchmark","measure":{"label":"Word error rate","unit":"percent","short":"WER %","decimals":2,"lowerIsBetter":true},"dimensions":[{"key":"engine","label":"Engine","values":["Parakeet TDT v3","Whisper large-v3-turbo","Apple SpeechTranscriber"]},{"key":"maker","label":"Maker"},{"key":"language","label":"Language","values":["English","German","French","Spanish","Italian","Portuguese","Russian","Ukrainian","Croatian","Slovenian","Latvian","Lithuanian","Estonian","Finnish","Swedish","Danish","Dutch","Polish","Czech","Slovak","Hungarian","Romanian","Bulgarian","Greek","Maltese"]}],"include":"Our own measurement: each engine transcribes the same 100 recordings per language from the FLEURS test set (Google, 2022), chosen by a fixed rule before any engine ran, on one Mac with nothing sent over the network. Every engine gets the language it is asked to recognize.","exclude":"Cloud services (they cannot run on the device) and languages an engine does not support.","method":"The recordings are the FLEURS test split at a pinned revision, 100 per language at evenly spaced positions. Each engine's text and FLEURS's written sentence are normalized the same way (Unicode NFKC, lowercase, words as runs of letters and digits) and compared word by word; the word error rate is the total of substitutions, deletions and insertions over the total of reference words. Every transcript, every timing and the SHA-256 of every recording are published in the raw record, so anyone can recompute every number.","limitations":"Read sentences from Wikipedia in quiet conditions are easier than dictation, conversation or noise. Numbers and spelling variants are not normalized, which counts '25' against 'twenty-five' as an error for every engine alike. One Mac, one run; we make Yap, which uses Parakeet, so we publish everything needed to check us.","sources":0,"updated":"2026-10-19","license":"https://creativecommons.org/licenses/by/4.0/","credit":"Yap (2026). On-device speech recognition word error rates on FLEURS [Data set]. https://yapapp.cc/data/on-device-asr-benchmark"},"rows":[{"row":"Parakeet TDT v3, Italian","value":2.52,"low":null,"high":null,"as_written":"2.52","unit":"percent","engine":"Parakeet TDT v3","maker":"NVIDIA","language":"Italian","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Parakeet TDT 0.6B v3 had a word error rate of 2.52 percent on 100 Italian recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-parakeet-it-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Parakeet TDT v3, Spanish","value":2.79,"low":null,"high":null,"as_written":"2.79","unit":"percent","engine":"Parakeet TDT v3","maker":"NVIDIA","language":"Spanish","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Parakeet TDT 0.6B v3 had a word error rate of 2.79 percent on 100 Spanish recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-parakeet-es-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Parakeet TDT v3, Portuguese","value":3.62,"low":null,"high":null,"as_written":"3.62","unit":"percent","engine":"Parakeet TDT v3","maker":"NVIDIA","language":"Portuguese","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Parakeet TDT 0.6B v3 had a word error rate of 3.62 percent on 100 Portuguese recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-parakeet-pt-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Parakeet TDT v3, English","value":4.48,"low":null,"high":null,"as_written":"4.48","unit":"percent","engine":"Parakeet TDT v3","maker":"NVIDIA","language":"English","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Parakeet TDT 0.6B v3 had a word error rate of 4.48 percent on 100 English recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-parakeet-en-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Parakeet TDT v3, Russian","value":4.74,"low":null,"high":null,"as_written":"4.74","unit":"percent","engine":"Parakeet TDT v3","maker":"NVIDIA","language":"Russian","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Parakeet TDT 0.6B v3 had a word error rate of 4.74 percent on 100 Russian recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-parakeet-ru-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Parakeet TDT v3, German","value":4.84,"low":null,"high":null,"as_written":"4.84","unit":"percent","engine":"Parakeet TDT v3","maker":"NVIDIA","language":"German","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Parakeet TDT 0.6B v3 had a word error rate of 4.84 percent on 100 German recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-parakeet-de-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Parakeet TDT v3, French","value":5.29,"low":null,"high":null,"as_written":"5.29","unit":"percent","engine":"Parakeet TDT v3","maker":"NVIDIA","language":"French","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Parakeet TDT 0.6B v3 had a word error rate of 5.29 percent on 100 French recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-parakeet-fr-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Parakeet TDT v3, Ukrainian","value":5.65,"low":null,"high":null,"as_written":"5.65","unit":"percent","engine":"Parakeet TDT v3","maker":"NVIDIA","language":"Ukrainian","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Parakeet TDT 0.6B v3 had a word error rate of 5.65 percent on 100 Ukrainian recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-parakeet-uk-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Parakeet TDT v3, Slovenian","value":19.83,"low":null,"high":null,"as_written":"19.83","unit":"percent","engine":"Parakeet TDT v3","maker":"NVIDIA","language":"Slovenian","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Parakeet TDT 0.6B v3 had a word error rate of 19.83 percent on 100 Slovenian recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-parakeet-sl-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Whisper large-v3-turbo, Spanish","value":2.53,"low":null,"high":null,"as_written":"2.53","unit":"percent","engine":"Whisper large-v3-turbo","maker":"OpenAI","language":"Spanish","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Whisper large-v3-turbo had a word error rate of 2.53 percent on 100 Spanish recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-whisper-es-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Whisper large-v3-turbo, Italian","value":2.73,"low":null,"high":null,"as_written":"2.73","unit":"percent","engine":"Whisper large-v3-turbo","maker":"OpenAI","language":"Italian","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Whisper large-v3-turbo had a word error rate of 2.73 percent on 100 Italian recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-whisper-it-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Whisper large-v3-turbo, Portuguese","value":3.28,"low":null,"high":null,"as_written":"3.28","unit":"percent","engine":"Whisper large-v3-turbo","maker":"OpenAI","language":"Portuguese","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Whisper large-v3-turbo had a word error rate of 3.28 percent on 100 Portuguese recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-whisper-pt-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Whisper large-v3-turbo, English","value":3.92,"low":null,"high":null,"as_written":"3.92","unit":"percent","engine":"Whisper large-v3-turbo","maker":"OpenAI","language":"English","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Whisper large-v3-turbo had a word error rate of 3.92 percent on 100 English recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-whisper-en-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Whisper large-v3-turbo, Russian","value":4.59,"low":null,"high":null,"as_written":"4.59","unit":"percent","engine":"Whisper large-v3-turbo","maker":"OpenAI","language":"Russian","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Whisper large-v3-turbo had a word error rate of 4.59 percent on 100 Russian recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-whisper-ru-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Whisper large-v3-turbo, German","value":5.08,"low":null,"high":null,"as_written":"5.08","unit":"percent","engine":"Whisper large-v3-turbo","maker":"OpenAI","language":"German","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Whisper large-v3-turbo had a word error rate of 5.08 percent on 100 German recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-whisper-de-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Whisper large-v3-turbo, French","value":6.62,"low":null,"high":null,"as_written":"6.62","unit":"percent","engine":"Whisper large-v3-turbo","maker":"OpenAI","language":"French","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Whisper large-v3-turbo had a word error rate of 6.62 percent on 100 French recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-whisper-fr-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Whisper large-v3-turbo, Ukrainian","value":6.83,"low":null,"high":null,"as_written":"6.83","unit":"percent","engine":"Whisper large-v3-turbo","maker":"OpenAI","language":"Ukrainian","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Whisper large-v3-turbo had a word error rate of 6.83 percent on 100 Ukrainian recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-whisper-uk-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Whisper large-v3-turbo, Croatian","value":11.34,"low":null,"high":null,"as_written":"11.34","unit":"percent","engine":"Whisper large-v3-turbo","maker":"OpenAI","language":"Croatian","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Whisper large-v3-turbo had a word error rate of 11.34 percent on 100 Croatian recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-whisper-hr-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Whisper large-v3-turbo, Latvian","value":18.64,"low":null,"high":null,"as_written":"18.64","unit":"percent","engine":"Whisper large-v3-turbo","maker":"OpenAI","language":"Latvian","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Whisper large-v3-turbo had a word error rate of 18.64 percent on 100 Latvian recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-whisper-lv-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Whisper large-v3-turbo, Slovenian","value":23.51,"low":null,"high":null,"as_written":"23.51","unit":"percent","engine":"Whisper large-v3-turbo","maker":"OpenAI","language":"Slovenian","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Whisper large-v3-turbo had a word error rate of 23.51 percent on 100 Slovenian recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-whisper-sl-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Apple SpeechTranscriber, Italian","value":4.15,"low":null,"high":null,"as_written":"4.15","unit":"percent","engine":"Apple SpeechTranscriber","maker":"Apple","language":"Italian","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Apple SpeechTranscriber had a word error rate of 4.15 percent on 100 Italian recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-apple-it-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Apple SpeechTranscriber, Spanish","value":4.3,"low":null,"high":null,"as_written":"4.3","unit":"percent","engine":"Apple SpeechTranscriber","maker":"Apple","language":"Spanish","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Apple SpeechTranscriber had a word error rate of 4.30 percent on 100 Spanish recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-apple-es-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Apple SpeechTranscriber, French","value":6.59,"low":null,"high":null,"as_written":"6.59","unit":"percent","engine":"Apple SpeechTranscriber","maker":"Apple","language":"French","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Apple SpeechTranscriber had a word error rate of 6.59 percent on 100 French recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-apple-fr-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Apple SpeechTranscriber, English","value":6.73,"low":null,"high":null,"as_written":"6.73","unit":"percent","engine":"Apple SpeechTranscriber","maker":"Apple","language":"English","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Apple SpeechTranscriber had a word error rate of 6.73 percent on 100 English recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-apple-en-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Apple SpeechTranscriber, German","value":6.87,"low":null,"high":null,"as_written":"6.87","unit":"percent","engine":"Apple SpeechTranscriber","maker":"Apple","language":"German","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Apple SpeechTranscriber had a word error rate of 6.87 percent on 100 German recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-apple-de-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"},{"row":"Apple SpeechTranscriber, Portuguese","value":100,"low":null,"high":null,"as_written":"100","unit":"percent","engine":"Apple SpeechTranscriber","maker":"Apple","language":"Portuguese","conversion":null,"claim":"In our October 2026 benchmark on a Mac with an M2 Pro, Apple SpeechTranscriber had a word error rate of 100.00 percent on 100 Portuguese recordings from the FLEURS test set.","quote":null,"where_in_source":null,"fact_url":"https://yapapp.cc/facts/m-fleurs-2026-10-apple-pt-wer","source_title":null,"source_publisher":null,"source_year":null,"source_url":null,"source_doi":null,"checked":"2026-10-19"}]}