MegaScale-Data: Scaling Dataloader for Multisource Large Foundation Model Training

arXiv:2504.09844v4 Announce Type: replace-cross Abstract: Modern frameworks for training large foundation models (LFMs) employ dataloaders in a data-parallel manner, with each loader processing a disjoint subset of training data. When preparing data for LFM training that originates from multiple,...