When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models

The Counterintuitive Edge of 8-Bit Quantization

A new preprint from arXiv (2512.18934v2) presents findings that challenge a core assumption in continual learning for large language models: that lower precision quantization inevitably harms a model’s ability to learn new tasks without forgetting old ones. The researchers systematically compared FP16, INT8, and INT4 quantization across continual learning benchmarks, and the results are striking—INT8 quantization actually improved retention of previously learned knowledge compared to full FP16 precision, while INT4 suffered the expected degradation.

This is not a minor statistical blip. The paper demonstrates that 8-bit quantization introduces a beneficial regularization effect, dampening the magnitude of weight updates during fine-tuning on new tasks. In effect, the reduced numerical precision acts as a soft constraint that prevents the model from overfitting to new data at the expense of old knowledge. This is the opposite of what many practitioners would intuitively expect, where more precision is assumed to be strictly better for learning capacity.

Why This Matters for Deployment

The practical implications are significant. Continual learning is the Achilles’ heel of LLM deployment—models trained on static datasets become stale, but retraining from scratch is prohibitively expensive. The standard solution has been to use larger, higher-precision models and hope that rehearsal or regularization techniques suffice. This research suggests that for many use cases, the optimal strategy may be to intentionally quantize to 8-bit before beginning the continual learning process, rather than after.

For edge deployment and on-device learning, this is particularly relevant. INT8 quantization is already widely supported by modern hardware (NVIDIA GPUs with Tensor Cores, Apple Silicon, Qualcomm AI engines), and it offers a 2x memory reduction over FP16 with minimal accuracy loss on static tasks. The finding that it also improves continual learning means that edge models can be updated incrementally with less risk of catastrophic forgetting, all while using less memory and power.

Implications for AI Practitioners

First, the default assumption that higher precision is always better for fine-tuning should be revisited. Practitioners running continual learning pipelines should benchmark INT8 as a baseline, not just as a compression step after training. Second, the INT4 results serve as a cautionary tale—the benefits do not extend to extreme quantization. The sweet spot appears to be 8-bit, where the regularization effect is strong enough to help but not so strong that it cripples the model’s ability to learn new patterns.

Third, this work opens the door for more efficient continual learning strategies that combine quantization-aware training with rehearsal or knowledge distillation. If INT8 already provides a free regularization benefit, combining it with explicit forgetting-mitigation techniques could yield even better results.

Key Takeaways

• INT8 quantization improves continual learning by acting as a natural regularizer, reducing catastrophic forgetting compared to FP16 precision.
• The benefit is precision-specific: INT4 degrades performance, while INT8 offers a sweet spot between memory efficiency and learning stability.
• Practitioners should reconsider training pipelines: Quantizing to 8-bit before continual fine-tuning may outperform post-training quantization in long-term deployment scenarios.
• Edge and on-device learning gains the most: The combination of reduced memory footprint and improved forgetting resistance makes INT8 ideal for models that must be updated incrementally on resource-constrained hardware.