Preprint
Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of Deepfakes Circulated in 2024
Nuria Alina Chandra et al., arXiv, 2025
Read the originalRead on October 17, 2026
- What it is
- A benchmark of deepfakes that circulated online in 2024, including 56.5 hours of audio from 88 websites. It shows that open-source deepfake detectors perform much worse on real-world fakes than on older academic datasets.
- How much weight it carries
- Preprint, not yet confirmed as peer-reviewed in this check. It is built from real circulated media, which makes it more representative than lab datasets. The author list is partial.
Facts taken from this source
- Open-source audio deepfake detectors lost 48% of their AUC, a measure of accuracy, on the 2024 in-the-wild benchmark compared with earlier benchmarks.
“AUC decreasing by 50% for video, 48% for audio, and 45% for image models compared to previous benchmarks” (Abstract)
- Open-source image deepfake detectors lost 45% of their AUC on the 2024 in-the-wild benchmark compared with earlier benchmarks.
“AUC decreasing by 50% for video, 48% for audio, and 45% for image models compared to previous benchmarks” (Abstract)
- Open-source video deepfake detectors lost 50% of their AUC on the 2024 in-the-wild benchmark compared with earlier benchmarks.
“AUC decreasing by 50% for video, 48% for audio, and 45% for image models compared to previous benchmarks” (Abstract)