Can AI models leak what you dictated?

A trained AI model can reproduce some of the text it learned from, and text that appears in only one document of its training data can be reproduced too ¹. Research has shown this for language models and for speech models. If a dictation app trains on your audio, the words you dictated become part of that training data. A studied attack can then try to recover them. No study we found shows a commercial dictation app leaking users' dictation, so the risk is real in principle and unmeasured in practice.

Language models can repeat rare training text word for word

In a 2021 USENIX Security study, Nicholas Carlini and colleagues queried GPT-2 and recovered hundreds of verbatim text sequences from its training data ². The attack worked even on sequences that appeared in only one document in the training data ¹. A name or a phone number that appears in only one document of the data falls in that group.

A 2023 study tested a production system. The authors recovered over ten thousand examples from ChatGPT's training dataset, by prompting the model to diverge from its normal output ³. They put the query cost at $200 ⁴. The study shows how cheap extraction can be for a researcher, which is why training data is a privacy question and not only a technical one.

Speech models memorize recordings too

Speech models show the same behavior. A 2023 preprint from Google researchers states that neural networks can unintentionally memorize their training examples, which causes privacy concerns. It sets out to measure this in large speech recognition models ⁵. A 2024 study found that when a speech model has seen a recording during training, a noisy version of that recording can make the model transcribe its memorized sensitive transcript ⁶.

For dictation, this matters because a speech model trained on your recordings may hold the sensitive phrases you said, not only the sound. The studies used research models, so they do not tell you how a particular app behaves.

Removing names does not remove the words

One defense is de-identification, which means removing names and account IDs before training. It reduces one risk. It does not remove the words themselves. Only some of the six AI developers in a 2025 Stanford study say that they remove personal information before training ⁷. If an app's policy does not say how it de-identifies data, you cannot tell from the policy how well it works. Our guide on whether dictation apps train AI on your voice covers what de-identification does and does not protect.

It depends on whether your audio is used for training at all

The risk depends on four conditions. Each one can change the answer:

• Training on your audio. If the app never trains on your recordings, there is no training data to leak. Check the setting described in how to stop apps from training on your voice.
• Retention. Audio kept for longer gives an attacker more chances to find it. Read the policy's storage period, not only its training clause.
• What you say. Rare, identifying details, such as a client's name or an account number, are the kind of text the 2021 study could recover when it appeared in only one document ¹.
• Where recognition runs. On a Mac, general text Dictation is processed on your device and is not sent to Siri servers ⁸, so a server-side training pipeline does not receive that audio. The policy still matters for account and usage data.

What to do to lower the risk

1. Turn off any setting that uses your audio for training, and confirm that it is off on the account you use.
2. Read the storage period and the human review clause, because they set how long your words exist outside your device.
3. Avoid dictating account numbers and passwords. Those are the details you would least want a model to repeat.
4. Prefer recognition on your device when you can. Yap, which we make, runs recognition on your computer. Check its settings and policy as you would for any other app.

Our answer rests on research with specific models and attacks, so treat it as a description of the risk, not a verdict on any product. The related questions cover the next steps: whether Otter.ai uses your recordings to train its AI and whether dictation apps send your words to a language model. The voice profile entry explains a related case, where an app builds a model of your own voice. For the full comparison, see our guide on whether dictation apps train AI on your voice and the voice privacy overview.