Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression

arXiv:2406.11354v3 Announce Type: replace-cross Abstract: Humans can retain old knowledge while learning new information, but Large Language Models (LLMs) often suffer from catastrophic forgetting when post-pretrained or supervised fine-tuned (SFT) on domain-specific data. Moreover, for Multimodal...