Predicting Closed-Loop Performance of Latent World Models: Offline Checkpoint Selection for MPC and Model-Based RL Under Non-Markovian Rewards in LunarLander

This new paper from ArXiv tackles a practical, often-overlooked bottleneck in model-based reinforcement learning (MBRL) and Model Predictive Control (MPC): how to pick the best neural network checkpoint for a latent world model without running the full, expensive closed-loop system.

The Core Problem: Validation Loss is a Liar

The researchers directly address a frustrating reality for AI engineers. When training a latent world model—a neural network that learns the hidden dynamics of an environment—standard practice is to monitor validation loss (e.g., prediction error on held-out sequences). The assumption is that lower validation loss correlates with better downstream performance when the model is used for planning or control. This paper demonstrates that this assumption is often false, particularly in the LunarLander environment. A checkpoint with excellent multi-step prediction accuracy can fail catastrophically when used for MPC, while a checkpoint with slightly worse validation loss might achieve perfect landings.

The key insight is the "distribution shift" problem. A world model trained offline sees state transitions from a fixed dataset. When deployed online, the MPC controller actively seeks out novel states—often edge cases or failure modes—that the model was never trained on. A model that memorizes the training distribution well (low validation loss) may have poor generalization to these out-of-distribution states, leading to brittle control.

Why This Matters for AI Practitioners

This research is significant because it moves the field from "train and hope" to "diagnose and select." For any practitioner deploying learned models in safety-critical or real-time systems, the paper offers a concrete methodology: offline checkpoint selection using closed-loop proxies.

The authors propose using diagnostics like the "value prediction error" or the "disagreement" between an ensemble of models, rather than raw prediction loss. These metrics, computed purely from validation data, can predict which checkpoint will yield stable, high-reward trajectories when used for planning. This is a direct cost-saver. Instead of running hundreds of expensive rollouts on a robot or simulator to test every saved checkpoint, an engineer can run a lightweight validation script that identifies the top candidates.

Furthermore, the focus on "non-Markovian rewards" (rewards that depend on history, not just the current state) is a nod to real-world complexity. Many tasks—like a robot needing to maintain a steady pace or a drone avoiding a no-fly zone—cannot be solved with simple, immediate reward signals. The paper’s framework shows that world models can still be effectively selected for these harder tasks.

Implications for the Model-Based RL Pipeline

This work reinforces a trend toward "robustification" in learned control. The era of simply minimizing a loss function and deploying is ending. Practitioners must now treat world model selection as a distinct engineering phase. The paper implicitly argues for a two-step process: (1) train a diverse set of checkpoints, and (2) use a targeted offline diagnostic (like ensemble disagreement) to pick the one that will generalize best under closed-loop stress.

For teams building autonomous systems, this is a warning against over-reliance on standard validation curves. It also provides a practical tool: if your MPC controller is failing unexpectedly, the problem may not be the training data or the architecture, but the specific parameter snapshot you chose to deploy.

Key Takeaways

• Validation loss is a poor predictor of closed-loop control performance. A world model with lower prediction error can fail during deployment due to distribution shift.
• Offline diagnostics like ensemble disagreement or value prediction error can reliably rank checkpoints for downstream MPC and MBRL tasks, saving significant computational cost.
• The findings apply to non-Markovian reward structures, making the method relevant for complex, real-world tasks where rewards depend on history.
• Practitioners should treat checkpoint selection as a distinct validation phase, using closed-loop proxies rather than relying solely on open-loop prediction metrics.