GWT: Scalable Optimizer State Compression for Large Language Model Training

arXiv:2501.07237v5 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have demonstrated exceptional capabilities across diverse natural language processing benchmarks. However, the escalating scale of model parameters imposes prohibitive memory overheads during training, especially...