Where to Place the Query? Unveiling and Mitigating Positional Bias in In-Context Learning for Diffusion LLMs via Decoding Dynamics

A New Blind Spot in Diffusion LLMs: Positional Bias in In-Context Learning

A recent preprint (arXiv:2606.19349v1) tackles a previously underexplored problem: how In-Context Learning (ICL) behaves in Diffusion Large Language Models (dLLMs). While ICL has been thoroughly investigated in autoregressive (AR) models like GPT-4, its dynamics in diffusion-based architectures—which generate text by iteratively denoising a latent representation rather than predicting one token at a time—remain poorly understood. The researchers identify a specific failure mode: positional bias, where the model’s ability to follow in-context examples depends heavily on where those examples appear in the input sequence.

What the Research Reveals

The core finding is that dLLMs, unlike AR models constrained by unidirectional causal masking, exhibit a different sensitivity to example ordering. In AR models, position bias is often tied to recency effects (later examples matter more) or primacy effects (early examples set the pattern). The paper shows that dLLMs suffer from a distinct form of bias rooted in their decoding dynamics—the iterative refinement process that gradually shapes the output. Specifically, the model’s attention patterns during early denoising steps can disproportionately favor examples placed at certain positions, leading to inconsistent ICL performance even when the same examples are simply reordered.

The authors propose mitigation strategies that adjust how the model attends to different positions during the decoding process, effectively re-weighting the influence of in-context examples. This is not a trivial fix; it requires modifying the diffusion sampling procedure itself, not just the input formatting.

Why This Matters

This research is significant for several reasons. First, dLLMs are emerging as a competitive alternative to AR models, promising faster generation and better handling of long-range dependencies. But if their ICL capabilities are brittle and position-dependent, practitioners cannot rely on them for few-shot tasks without careful prompt engineering. Second, the finding highlights that lessons from AR models do not transfer directly to diffusion architectures. The field has spent years optimizing ICL for AR models—prompt ordering, example selection, formatting tricks—and this paper suggests we may need to start from scratch for dLLMs.

For AI practitioners, the immediate implication is caution. If you are experimenting with diffusion-based LLMs for few-shot classification, code generation, or instruction following, you cannot assume that a well-structured prompt for GPT-4 will work equally well. The positional bias may cause erratic performance, and naive reordering of examples could yield unpredictable results.

Implications for AI Practitioners

• Prompt engineering for dLLMs is not yet mature. Standard ICL heuristics (e.g., placing the most relevant example last) may not apply. Practitioners should test multiple orderings and consider dynamic attention adjustment.
• Model selection requires new benchmarks. Existing ICL evaluation suites designed for AR models may not capture dLLM-specific biases. Teams evaluating dLLMs should include position-randomization tests.
• Decoding-time interventions matter. The proposed mitigation—modifying attention during diffusion steps—points to a broader trend: improving model behavior by altering the generation process, not just the training data. This is a more complex but potentially more powerful lever than prompt tuning.

Key Takeaways

• Diffusion LLMs exhibit a unique positional bias in in-context learning, distinct from the recency/primacy effects seen in autoregressive models, stemming from their iterative decoding dynamics.
• Standard prompt engineering techniques for AR models do not reliably transfer to dLLMs; practitioners must re-validate ICL performance across multiple example orderings.
• Mitigating this bias requires modifying the diffusion sampling process (e.g., attention re-weighting), not just input formatting—a more technically involved but promising approach.
• The research underscores that dLLMs are not simply drop-in replacements for AR models; their unique architectural properties demand new evaluation protocols and deployment strategies.