Why Advanced Encoders Lag on Sparse Retrieval? The Answer and an Approach to Bridging Vocabulary Gaps

The Sparse Retrieval Paradox: Why Newer Encoders Fail Where BERT Succeeds

A new arXiv paper (2607.00004) has uncovered a surprising performance gap in learned sparse retrieval (LSR): advanced foundation models like ModernBERT, which dominate dense retrieval benchmarks, actually perform worse than the aging BERT-base when used for sparse retrieval tasks. The researchers identify the root cause as a vocabulary mismatch—newer models use tokenizers with smaller, more optimized vocabularies that lack the lexical coverage needed for effective sparse retrieval.

In LSR, models assign importance weights to individual terms from a fixed vocabulary, creating sparse vectors that mimic traditional inverted indexes. ModernBERT’s tokenizer, while efficient for dense representations, simply doesn’t include enough of the rare and domain-specific terms that BERT’s larger vocabulary captures. This creates a “vocabulary gap” where important query terms have no corresponding dimension in the sparse vector space.

Why This Matters

The finding challenges the assumption that “better” foundation models automatically improve all downstream tasks. Sparse retrieval remains critical for applications requiring interpretability (you can see exactly which terms matched), low-latency search (sparse vectors are naturally compressible), and hybrid systems that combine dense and sparse signals. If practitioners blindly upgrade to ModernBERT for their retrieval pipelines, they risk degrading search quality precisely where sparse methods excel.

The paper also highlights a fundamental tension: dense retrieval benefits from compact, semantically rich representations, while sparse retrieval depends on broad lexical coverage. These are competing design goals, and the industry’s push toward smaller, more efficient tokenizers may inadvertently harm lexical retrieval capabilities.

Implications for AI Practitioners

First, evaluate retrieval tasks independently. Don’t assume that a model’s dense retrieval performance predicts its sparse retrieval quality. Second, vocabulary size matters for sparse methods. When building LSR systems, consider models with larger vocabularies or explicitly augment tokenizers with domain-specific terms. Third, hybrid retrieval systems need careful tuning. If you combine dense and sparse scores, the sparse component may degrade with newer models, requiring recalibration of fusion weights.

The paper also proposes a bridging approach—likely involving vocabulary expansion or term-level knowledge distillation—though full details require reading the preprint. For teams deploying production search, this is a timely reminder that architectural “improvements” are not universal.

Key Takeaways

• ModernBERT and similar advanced encoders underperform BERT-base in learned sparse retrieval due to vocabulary gaps from smaller, optimized tokenizers.
• Dense and sparse retrieval have competing requirements: semantic richness vs. lexical coverage—newer models optimize for the former at the expense of the latter.
• Practitioners should benchmark sparse retrieval separately when upgrading foundation models, and consider vocabulary augmentation for LSR tasks.
• Hybrid retrieval systems may need recalibration when switching to newer encoders, as the sparse component’s contribution can degrade unexpectedly.