arxiv-2405-17830 SRC-0534 · paperMore Than Catastrophic Forgetting: Integrating General Capabilities For Domain-Specific LLMs
Chengyuan Liu, Shihang Wang, Yangyang Kang, Lizhi Qing, Fubang Zhao, Changlong Sun
Created: 2024-05-28 · Ingested: 2026-09-23
https://arxiv.org/abs/2405.17830(opens in a new tab)The performance on general tasks decreases after Large Language Models (LLMs) are fine-tuned on domain-specific tasks, the phenomenon is known as Catastrophic Forgetting (CF). However, this paper presents a further challenge for real application of domain-specific LLMs beyond CF, called General Capabilities Integration (GCI), which necessitates the integration of both the general capabilities and domain knowledge within a single instance. The objective of GCI is not merely to retain previously acquired general capabilities alongside new domain knowledge, but to harmonize and utilize both sets
In brief
Fixing catastrophic forgetting does not make a fine-tuned model actually use its general skills on domain tasks. After legal fine-tuning, models still produce complete reasoning chains and keep instruction-following, yet fail to combine that behaviour with the injected legal knowledge, which the authors name General Capabilities Integration (GCI).
They build 3 train/test task groups in Chinese law: Article QA then reasoning and calculation; Article QA then ethic-attack and contaminated-query dialogues; legal information extraction then event summary and MRC. Datasets come from CAIL, legal consulting sites, court verdicts, and GPT-4 expansion of hand-written seeds. Baselines include LoRA SFT, wise-ft, L1/L2 and KL regularisation, DAS, MixTraining, and MixDA*. Their method, ALoRA, adds a multi-head attention module over LoRA so the adapter reads preceding tokens.
6 baselines reach 100 on reasoning "Chain" while BLEU_R stays low. DAS raises training-task scores but leaves test tasks near the untuned chat model, with Score_E of 2 versus 47.94 for ALoRA. ALoRA leads averages at 19.50 and 20.73 on reasoning/calculation.
Evidence is measured with matched baselines and an ablation that raises LoRA rank from 8 to 16 to control trainable parameters; removing attention costs nearly 10. Gains are often under 1 point, scoring for safety tasks is GPT-4-based, and only the legal domain and 1 main backbone are tested. ALoRA also struggles on long-verdict summarisation.
Treat forgetting benchmarks as insufficient evidence that a domain model retained usable general skills.
Written from the abstract by claude-opus-5 on 2026-10-24, with every figure checked against it. Not a substitute for the paper.
Referenced by
Claims in this catalog that draw on this source, and whether as support or counterpoint.
No claims cite this yet.