InfiniPipe: Elastic Pipeline Parallelism for Efficient Variable-Length Long-Context LLM Training

arXiv:2509.21275v3 Announce Type: replace-cross Abstract: Long context training is crucial for LLM's context extension. Existing schemes, such as sequence parallelism, incur substantial communication overhead. Pipeline parallelism (PP) reduces this cost, but its effectiveness hinges on partitioning...