# Catastrophic Forgetting

Losing previously acquired general capability while training on new narrow data — the central risk of domain adaptation.

**Rests on:** Pre-Trained Model Adaptation, Training Dynamics

*LLMs and Generative AI · Fine-Tuning*


## Why learn this

Catastrophic forgetting is why most attempts to teach a trained language model new skills silently ruin its existing knowledge. Imagine you fine-tune a model on your company's support tickets for a week, and it immediately starts forgetting how to answer general questions like 'what is the weather?' or 'write a poem.' You have hit this wall. Understanding this phenomenon tells you when full model adjustment is a mistake, why lightweight adaptation methods exist, and why mixing broad, everyday data into your new training set is the cheapest insurance you will ever buy. It is the concept that connects model adaptation to the everyday decision every developer makes: do I reshape the whole brain, or do I attach a small, temporary note? Master this, and you will never be the person who wonders why their model got stupider after trying to 'improve' it.


## The intuition

Imagine you have a brilliant generalist employee who knows a little about everything — customer service, legal, finance, cooking. Now you want them to specialize in your company's loan approval process. You send them on a two-week intensive training course, where all they see, all day, is loan applications: hundreds of thousands of them. By the end, they are a loan-approval expert. But when you casually ask them a question about cooking, they hesitate. Ask them about the legal basics they used to know, and they give a shaky answer. The intensive training overwrote some of what they knew. It did not happen because the course was bad — it happened because the course was so focused that the old knowledge had no chance to be practiced, and the brain (or the model) kept only what it saw recently.


## An analogy

Think of a musician who has spent years mastering classical piano. The repertoire is vast — Bach, Beethoven, Chopin — all stored in muscle memory. Now they decide to learn jazz. They hire a jazz coach who drills them for weeks, eight hours a day, on nothing but syncopated chords and blues scales. Jazz is hard and demands new finger patterns that fight the classical habits. After a month of this, they sit down to play a Chopin nocturne and find their fingers keep sliding into jazz phrasings. The classical pieces are not gone entirely, but they are badly degraded, because the intense jazz practice rewired the same neural pathways. The analogy breaks down in two places: a human musician can deliberately rehearse old pieces to preserve them (a model cannot), and a human's decline is gradual (a model's loss can be sudden). The remedy in both cases is the same: keep practicing the old material alongside the new.


## Definition

Catastrophic forgetting is the phenomenon where a neural network, while being trained on new data for a specific task, loses capabilities it previously had on older, broader data, because the weight updates that fit the new data overwrite the weights that encoded the old knowledge.


## How it connects

You are studying this as part of fine-tuning, which is taking a pretrained model and continuing training on new data. Catastrophic forgetting is the central risk of that process. You have already learned about pretraining (how the model got its general knowledge) and large language models. This concept connects directly to LoRA and QLoRA, which you will study next: those methods freeze the base weights and train small adapters, precisely to avoid catastrophic forgetting. It also connects to continued pre-training and instruction tuning, where the risk is real but the data is often close enough to the original that the damage is smaller.


## The mathematics

The mathematics here is simple arithmetic, not calculus. A neural network's knowledge is stored in its weights — numbers that multiply the inputs. When the model learns, those weights get adjusted a tiny bit at a time. The adjustment is: for each weight, calculate a small correction based on how wrong the model's answer was on the new data, and subtract a fraction of that correction from the weight. If the new data is very different from the old, the corrections are large and every weight moves a lot. The old knowledge is stored in the old values of those weights. When they move far enough, the old knowledge is gone. The relevant branch of mathematics is arithmetic and a little bit of algebra — but you only need to add, multiply, and compare numbers to see the effect. Forgetting is not a special formula; it is a consequence of changing many numbers at once.

The update rule for each weight during training is: w_new = w_old - learning_rate * (correction). The correction is usually a gradient, but you do not need to know what a gradient is to understand the risk: the larger the correction, the more the weight moves, and the more old knowledge is lost. Forgetting is not a separate equation; it is the accumulation of these updates. A common measurement is how much performance on an old task drops after training on a new task: Forgetting = (performance_on_old_task_before) - (performance_on_old_task_after).


## A worked example

```
Suppose a tiny model has just two weights, w1 and w2, that together encode its knowledge. For general knowledge, the ideal values are w1 = 2.0 and w2 = 0.5. Now we fine-tune on loan data. After one batch of training, each weight is updated by a correction. For aggressive fine-tuning, the corrections are large: w1 changes by -1.5, w2 changes by +1.0.

Step 1: Old general weights: w1 = 2.0, w2 = 0.5   # These values give good answers on general questions.
Step 2: Update for w1: w1_new = 2.0 - 1.5 = 0.5   # The correction -1.5 is large because the new data demands a very different value.
Step 3: Update for w2: w2_new = 0.5 + 1.0 = 1.5   # The correction +1.0 also moves w2 far from its old value.
Step 4: Difference for w1: |2.0 - 0.5| = 1.5   # The old weight has moved by 1.5, which is 75% of its original value.
Step 5: Difference for w2: |0.5 - 1.5| = 1.0   # The old weight has moved by 1.0, which is 200% of its original value.

Result: Both weights have changed drastically. The model now answers new-data questions well, but the old general knowledge is essentially erased because the weights no longer hold the values that produced it. A small change in one weight is harmless; a large change in many weights is catastrophic.
```


## From scratch, in Python

```python

# Simulate catastrophic forgetting with a tiny model

# A 'model' is just a list of weights (numbers).
# We'll use two weights to keep it visible.

weights = [2.0, 0.5]  # general knowledge: good values for many tasks

# New data demands different weights.
# In real training, these corrections are computed from the loss.
# Here we pretend the new data 'wants' w1=0.5, w2=1.5.

target_weights = [0.5, 1.5]

# Learning rate: how big a step we take toward the target.
# With a high learning rate, the step is large and old knowledge is destroyed.

learning_rate = 0.9

print("Before training:", weights)

# One training step: move each weight partway toward the target.

for i in range(len(weights)):
    correction = target_weights[i] - weights[i]
    weights[i] = weights[i] + learning_rate * correction

print("After one aggressive step:", weights)

# Now test on an 'old task'. A simple check: how close are the weights
# to what the old task needed? (1.0 = perfect, lower is worse.)

def old_task_score(w):
    # Higher is better; we use a made-up formula.
    return 1.0 - abs(w[0] - 2.0) / 2.0  # only checks w1

score_before = 1.0 - abs(2.0 - 2.0) / 2.0  # 1.0
score_after = old_task_score(weights)       # weights[0]=0.5

print("Old task score before:", score_before)
print("Old task score after:", score_after)
print("Forgetting amount:", score_before - score_after)

# Now simulate the standard mitigation: mix in some general data.
# Instead of moving fully toward the new target, we move only 30% of the way.

weights_mitigated = [2.0, 0.5]
learning_rate_small = 0.3

for i in range(len(weights_mitigated)):
    correction = target_weights[i] - weights_mitigated[i]
    weights_mitigated[i] = weights_mitigated[i] + learning_rate_small * correction

print("Mitigated weights:", weights_mitigated)
print("Old task score after mitigation:", old_task_score(weights_mitigated))

```


## What the code does

- `weights = [2.0, 0.5]` creates a list of two numbers. A list is a Python container that keeps values in order. Here, index 0 holds w1, index 1 holds w2.
- `target_weights = [0.5, 1.5]` is a second list, representing what the new data 'wants' the weights to be.
- `learning_rate = 0.9` is a number that controls the step size. In real training it is small (e.g., 0.001), but we use a large one to make the effect obvious.
- The `for i in range(len(weights)):` loop goes through each index from 0 up to (but not including) the length of the list. Inside, `correction` is the difference between target and current, and the weight is updated. This is the same arithmetic you did by hand.
- The function `old_task_score(w)` takes a list `w` and returns a number. It shows how well the model would do on an old task, purely based on how close w1 is to its ideal value. `abs` gives the absolute value (distance from zero).
- The mitigation uses a smaller `learning_rate_small` (0.3) which is analogous to mixing in general data: the update is gentler, so old knowledge is disturbed less.


## Common misconceptions

**Forgetting is gradual and slow** — If I fine-tune carefully, the old knowledge will fade slowly, not suddenly.

In a neural network, the loss of old knowledge is often abrupt. A single large update on very different data can push weights past the point where old patterns are recognizable, causing a sudden cliff in performance. In fact: Forgetting can be sudden and total if the new data is very different. The standard mitigation is to keep the updates small and to mix general data in.

**Fine-tuning only adds new knowledge** — Fine-tuning adds new skills on top of old ones, like learning a new language after English.

Fine-tuning changes the same weights that encode old knowledge. There is no separate storage. The new data literally overwrites old values. In fact: Fine-tuning is a trade: you trade some general ability for specialisation. The art is to minimise the trade by controlling the updates.

**More training data always helps** — If I train my model for longer on domain data, it will just get better at the domain without losing anything.

Longer training on the same narrow data makes the forgetting worse, because the weights keep moving further from their original values. In fact: More data helps only if it is diverse. The standard practice is to limit training epochs or mix in general data.

**Benchmarks will catch forgetting** — As long as my model scores well on my eval set, it's fine.

Your eval set is probably as narrow as your training data. It won't test general ability, so the forgetting goes unnoticed. In fact: You need a separate general-evaluation set that includes tasks unrelated to your domain, to detect forgetting.

**Parameter-efficient methods eliminate forgetting entirely** — LoRA freezes the base weights, so forgetting is completely impossible.

The adapter still sits on top of the base, and it can influence the output to ignore base knowledge. The base weights are safe, but the combined model can still forget if the adapter is strong enough. In fact: PEFT reduces forgetting by keeping the base intact, but the adapter's output can still dominate. The risk is lower, not zero.


## In short

Catastrophic forgetting is the loss of general capability when a model is trained on narrow new data. It happens because the same weights encode both old and new knowledge, and updating them for the new task moves them away from the old values. The standard mitigation is to mix general data into the fine-tuning set, use smaller learning rates, or use parameter-efficient methods like LoRA. Always evaluate on general tasks, not just your domain, to catch forgetting before it ships.


## Where this is used

Catastrophic forgetting is the fundamental reason almost no serious AI system does full fine-tuning. Here is where it bites, and what developers actually do.

**Large language models (LLMs):** When a company like yours fine-tunes an LLM on their own data — say, customer support transcripts — the model is usually trained on millions of examples of that narrow kind. Without mitigation, the model would forget how to write poetry, answer general trivia, or follow basic instructions. The standard practice is to mix a large fraction of general-purpose instruction data into the fine-tuning set — often 50% or more. Developers use libraries like Hugging Face's Transformers and set the training data to include a mix of general and domain-specific examples. If they skip this, they see the model's perplexity on general benchmarks skyrocket, and the model starts generating nonsense on unrelated prompts.

**Small models on devices (e.g., a phone or a Raspberry Pi):** TinyML models for keyword spotting or sensor classification are often adapted from a general pretrained model. The risk is that fine-tuning on the new keyword set erases the ability to recognise the original set. Developers use the same mitigation — keeping a portion of the original keyword data in the training set — and often add a regularization term that penalises large weight changes (called elastic weight consolidation, but the idea is the same: don't move the weights too far). On-device, the model cannot be huge, so the forgetting is proportionally worse because every weight matters more.

**Retrieval-augmented generation (RAG):** RAG systems don't fine-tune the LLM at all — they retrieve documents and stuff them into the prompt. So catastrophic forgetting is mostly avoided. However, the embedding model (used to search documents) might be fine-tuned on a narrow domain, and that can cause it to forget general semantics. Developers often freeze the embedding model and only train a small adapter on top, or use a general embedding model and rely on a hybrid search.

**Agentic systems:** Agents plan, call tools, and act in loops. If you fine-tune a base LLM to be better at tool calling, you might lose its general reasoning ability. The mitigation is to fine-tune on a balanced dataset that includes both tool-calling and general instruction-following examples. Developers also use parameter-efficient methods like LoRA, which train a small set of extra numbers rather than touching the base weights, so the base knowledge is preserved. LoRA is the standard answer — the library notes say 'Parameter-efficient methods reduce but do not eliminate it'.

**Everyday applied machine learning:** In computer vision, a model pretrained on ImageNet is often fine-tuned on a small dataset of, say, X-ray images. Without mixing in some ImageNet data, the model forgets generic features like edges and corners, and then overfits to the X-ray domain. Similarly, in fraud detection, a model trained on credit card transactions might forget how to detect a different type of fraud if trained exclusively on the new type. The developer's job is to blend the old and new data, or to use a 'rehearsal' technique where the model is periodically re-exposed to old examples.

In practice, a developer reaches for **PyTorch or TensorFlow** for full fine-tuning, but more commonly **Hugging Face PEFT** with LoRA. They write code like this:

```python
from peft import LoraConfig, get_peft_model
config = LoraConfig(r=8, lora_alpha=16, target_modules=["q_proj", "v_proj"])
model = get_peft_model(base_model, config)
```

This freezes the base weights and adds tiny trainable matrices. The base model never changes, so catastrophic forgetting is largely prevented. What breaks when done badly: your model becomes a specialist that can't do anything else, benchmarks on general tasks collapse, and customers complain the assistant 'became dumb'. The fix is always the same: mix in general data, use a smaller learning rate, or switch to PEFT.


---

Read this on the web: https://basavapurushottam.com/learn-ai/catastrophic-forgetting
