WhiADD: Semantic-Acoustic Fusion for Robust Audio Deepfake Detection

## One-line Summary Whisper-based semantic features fused with acoustic embeddings via cross-attention significantly improve deepfake audio detection robustness across unseen codecs. ## Problem Existing deepfake audio detectors rely purely on low-level acoustic features (MFCCs, spectrograms) and collapse when faced with unseen codec-based TTS/VC systems. Semantic content — what was said — is underexplored as a complementary signal. ## Method - **Semantic branch:** Whisper encoder extracts high-level text-semantic representations from raw audio. - **Acoustic branch:** Standard front-end (e.g., RawNet2-style) extracts low-level acoustic artifacts. - **Fusion:** Gated Cross-Attention merges the two streams; the gate learns when to trust semantics vs. acoustics. - **Training:** Sharpness-Aware Minimization (SAM) stabilizes training and improves generalization to unseen conditions. ## Key Results - EER drops from 3.1% → 2.3% on ASVspoof2019 LA - Strong cross-corpus generalization on In-the-Wild dataset - Ablation shows both branches are necessary; removing semantics hurts most on codec-based fakes ## My Takeaways The core insight is that semantic consistency (e.g., a deepfake voice may have subtle mismatch between "what it sounds like" and "what Whisper thinks was said") carries forensic signal. This is promising for future work combining ASR confidence scores as a lightweight proxy. **Open question:** Does this break when the TTS system is also Whisper-aligned?