From Dispersion to Attraction: Spectral Dynamics of Hallucination Across Whisper Model Scales

This new research from arXiv introduces the Spectral Sensitivity Theorem, offering a novel mathematical framework for understanding why and how hallucinations emerge in large-scale automatic speech recognition (ASR) models like OpenAI’s Whisper. Rather than treating hallucinations as random errors, the authors model them as a predictable phase transition in the network’s spectral dynamics.

What Happened

The paper proposes that as model scale increases, the internal representation dynamics shift from a “dispersive” regime—where signal energy decays and noise is naturally filtered out—to an “attractor” regime. In this second phase, the network’s hidden states become overly self-reinforcing, locking onto spurious patterns that do not correspond to the input audio. Essentially, the model begins to “hear” what it expects to hear, rather than what is actually spoken. The theorem predicts a critical threshold in model size or depth beyond which hallucinations become not just more frequent, but qualitatively different: structured, persistent, and harder to detect.

The researchers validated this using controlled scaling experiments across multiple Whisper model variants (small, medium, large), observing that hallucination patterns indeed follow the predicted spectral bifurcation. The work provides a mathematical grounding for a phenomenon that practitioners have long observed empirically: bigger ASR models can be more confident in their errors.

Why It Matters

This is significant for three reasons. First, it moves the conversation about hallucinations from anecdotal observation to a testable, physics-inspired framework. If the spectral dynamics of a model can be analyzed pre-deployment, developers might predict its hallucination propensity without exhaustive red-teaming.

Second, it challenges the assumption that “more data and more parameters” will automatically reduce hallucinations. The theorem suggests that beyond a certain scale, the model’s internal dynamics can cause hallucinations as an emergent property of its architecture, not a data deficiency. This has direct implications for safety-critical applications like medical transcription, legal proceedings, or automated captioning.

Third, the work opens a path toward architectural interventions. If the attractor regime is the root cause, then modifying the spectral properties of the network—through regularization, skip connections, or spectral normalization—could suppress hallucinations more effectively than post-hoc filtering or confidence thresholds.

Implications for AI Practitioners

For engineers deploying Whisper or similar large ASR models, the key takeaway is that scale is a double-edged sword. A larger model may transcribe common phrases more accurately, but it may also hallucinate more convincingly when the audio is ambiguous or out-of-distribution. Practitioners should:

• Audit for spectral behavior during model selection, not just WER and CER.
• Consider hybrid approaches where a smaller, “dispersive” model flags uncertain segments for a larger model, rather than relying on a single large model end-to-end.
• Implement dynamic confidence thresholds that account for the attractor regime—hallucinated text often exhibits abnormally low spectral entropy, which could be detected in real-time.

Key Takeaways

• The Spectral Sensitivity Theorem mathematically predicts a phase transition in large ASR models where hallucinations shift from random noise to structured attractor states.
• Beyond a critical scale, model size can cause hallucinations rather than cure them, challenging the assumption that bigger is always safer.
• Practitioners should evaluate models for spectral dynamics and consider architectural mitigations (e.g., spectral normalization) rather than relying solely on data scaling or post-hoc filtering.
• Real-time detection of low-entropy attractor states could serve as a practical hallucination alarm for production ASR systems.