From Tokens to States: LLMs as a Special Case of World Models and the Continuous Path Beyond

What Happened

A new arXiv paper (2606.28127) challenges the prevailing dichotomy between large language models and world models. The authors argue that LLMs are not fundamentally separate from world models, but rather represent a special case—one that operates on discrete tokens rather than continuous state representations. The work directly engages with Yann LeCun’s 2022 argument that true general intelligence requires abandoning token-based prediction in favor of models that simulate continuous reality.

The paper proposes that the distinction is not categorical but architectural: both LLMs and world models aim to predict future states, but LLMs do so in a compressed, discrete token space. The authors then explore a “continuous path beyond” current LLM architectures, suggesting that future systems may bridge the gap by operating on continuous latent representations while retaining the learned knowledge from token-based training.

Why It Matters

This reframing has significant implications for how the AI community conceptualizes progress toward general intelligence. If LLMs are already a form of world model—just an impoverished one—then the path forward is not about abandoning them but about extending their representational capacity. The paper suggests that the tokenization bottleneck is the key limitation, not the underlying predictive objective.

For researchers, this means that efforts to build world models should not be seen as starting from scratch. Instead, they can leverage the vast knowledge encoded in existing LLMs and gradually transition those systems from discrete token prediction to continuous state simulation. This could accelerate progress toward embodied AI and robotics applications where discrete language alone is insufficient.

Implications for AI Practitioners

Architecture design: Practitioners working on multimodal or embodied AI should consider hybrid approaches that maintain an LLM’s learned weights while adding continuous processing pathways. The paper implies that fine-tuning on continuous representations may be more fruitful than building entirely new architectures. Evaluation metrics: Current benchmarks focused on token-level accuracy may miss the deeper question of whether models are learning genuine world dynamics. Practitioners should develop metrics that test for continuous state prediction, not just next-token plausibility. Training strategies: The paper suggests a phased approach: start with token-based pretraining for broad knowledge acquisition, then transition to continuous state training for deeper world understanding. This could reduce the compute requirements currently associated with training world models from scratch. Research direction: For those building AI agents, the key insight is that the gap between LLMs and world models is narrower than commonly assumed. The bottleneck is representational, not conceptual—meaning that solving continuous representation learning could unlock general intelligence capabilities from existing LLM foundations.

Key Takeaways

• LLMs and world models share a predictive objective; the difference lies in discrete versus continuous state representations, not fundamental capability.
• The tokenization bottleneck is the primary limitation preventing current LLMs from functioning as full world models.
• Practitioners should explore hybrid architectures that retain LLM knowledge while adding continuous processing pathways.
• The path to general intelligence may involve transitioning existing LLMs to continuous representations rather than building entirely new model classes.