# Approximate Unlearning

Efficiently reducing a record's influence without full retraining, accepting bounded rather than absolute guarantees.

**Rests on:** Utility Preservation

**Leads to:** Federated Unlearning

*Machine Learning · Machine Unlearning*


## Why learn this

Mastering approximate unlearning lets you understand how modern AI systems handle requests to delete personal data without rebuilding the entire system from scratch. This balance between speed and accuracy is essential for responding to privacy laws like the Right to Be Forgotten. It also helps you evaluate different technical strategies, such as those that guarantee complete deletion by preparing the data in advance, or those that adapt to data coming from many different sources at once.


## The intuition

Imagine a model is a giant book of facts, and each training record is a sentence that contributed to the writing. Full retraining means rewriting the entire book from scratch without that sentence — expensive and slow. Approximate unlearning is like using white-out to erase the sentence and correcting the immediate nearby words, hoping the story still makes sense. It is fast, but you might miss a few subtle references elsewhere. The key idea is to reduce the influence of a record without guaranteeing that it left no trace at all — you accept a small, bounded risk that the model still 'remembers' something.


## An analogy

Think of your favourite restaurant's recipe book. A customer complained about a specific spice, so you want to remove it from the recipe. Retraining from scratch would be rewriting the entire recipe book from memory, checking every dish. Approximate unlearning is like taking the recipe card, scratching out the spice, and adjusting the amounts of the other spices to compensate. You don't rewrite the whole book; you just tweak the one recipe. This works well if the spice was only in that recipe, but if it was also a secret ingredient in the sauce, you might miss it. And the chef's memory still tingles when they smell that spice — the influence isn't fully gone. That's the trade-off: speed and simplicity versus certainty.


## Definition

Approximate unlearning is a set of methods to reduce a specific training record's influence on a model without retraining from scratch, accepting that the result is not guaranteed to be identical to a model that never saw that record.


## How it connects

You already know that naive full retraining is the correctness baseline and cost benchmark for all unlearning (from your notes on 'Machine Unlearning'). Approximate unlearning is the cheaper alternative that trades away the guarantee. It contrasts directly with exact unlearning, which requires structuring training in advance — such as a method called SISA that splits data into separate groups so you can delete one group easily — to make complete deletion affordable. Approximate methods are also relevant to federated unlearning, where contributions arrive as updates and isolating any one client is hard; approximate correction from those updates is what makes it practical.


## The mathematics

The mathematics of approximate unlearning builds on calculus, specifically the idea of a derivative, which measures how much a small change in one thing affects another. Before we touch that, let's first understand what a function is: a machine that takes a number in and gives a number out. For a model, the inputs are data and the outputs are predictions; the model's weights are the knobs we tune to make good predictions. When we train a model, we adjust these knobs to minimize the error on the training data. Now, when we want to unlearn a record, we don't want to adjust all knobs from scratch — we just want to nudge the ones that were most influenced by that record. The derivative tells us how strongly each weight depends on that one record: it's the slope of the relationship between the record's presence and the weight's value. A steep slope means the weight reacts strongly to that record; a shallow slope means it barely cares. Approximate unlearning works by subtracting a small multiple of this slope from the weights, nudging them back toward what they would have been without the record. The size of the nudge is controlled by a number called the learning rate, which we choose carefully — too big and we overshoot, too small and we don't forget enough.

w_new = w_old - α * ∇_w L_record


## A worked example

```
Let's say we have a model with a single weight, w, that has a value of 10.0 after training. We want to 'forget' one training record. From analysis, we know the derivative of the loss with respect to that record is 2.0 — meaning that if that record were removed, the weight would naturally decrease. We choose a learning rate of 0.1.

Step 1: Derivative (slope) = 2.0   # This is the degree to which the record pulls the weight up.
Step 2: Learning rate = 0.1   # We only want to move the weight a small amount, not erase it entirely.
Step 3: Step size = learning rate × derivative = 0.1 × 2.0 = 0.2   # This is how much we will subtract from the weight.
Step 4: New weight = old weight − step size = 10.0 − 0.2 = 9.8   # We nudge the weight down, hoping to approximate the weight it would have had without the record.

Result: The weight is now 9.8, slightly lower than before, approximating the removal of the record's influence.
```


## From scratch, in Python

```python
# Plain Python: approximate unlearning by nudging a single weight

# Let's assume we have a trained model with one weight, stored as a number.
# The model is just a function: output = weight * input.
# We also have a training record (input, output) that we want to forget.

# Step 1: Define our things in plain Python
weight = 10.0  # the current weight (after training)
record_input = 3.0
record_output = 8.0  # what the model was trained to predict for this input

# Step 2: Compute the derivative of the loss for this record.
# Loss (squared error) = (output - weight * input)^2
# Derivative of loss with respect to weight = -2 * (output - weight * input) * input
# Let's compute it step by step.
prediction = weight * record_input
loss = (record_output - prediction) ** 2
# The derivative (slope) of the loss with respect to the weight:
derivative = -2 * (record_output - prediction) * record_input

print("Before unlearning: weight =", weight, "prediction =", prediction, "loss =", loss)

# Step 3: Set a learning rate (a small number that controls the step size)
learning_rate = 0.1

# Step 4: Update the weight by subtracting a small multiple of the derivative
weight = weight - learning_rate * derivative

# Step 5: See the new weight and the new loss on this record
new_prediction = weight * record_input
new_loss = (record_output - new_prediction) ** 2

print("After unlearning: weight =", weight, "prediction =", new_prediction, "loss =", new_loss)
print("The model now pays less attention to this record.")
```


## The same thing, with a framework

The above is the core arithmetic. In practice, you'd use a library like PyTorch to handle large models. Here's how you'd do the same nudge using tensors:

```python
import torch

# Suppose the model is a simple linear layer: output = weight * input (plus bias, but we ignore it)
weight = torch.tensor([10.0], requires_grad=True)

# Training record
record_input = torch.tensor([3.0])
record_output = torch.tensor([8.0])

# Compute loss (mean squared error)
loss = (record_output - weight * record_input) ** 2

# Compute gradient of loss w.r.t. weight
loss.backward()

# Gradient is stored in weight.grad
grad = weight.grad.item()  # in our example, it should be -2 * (8 - 10*3) * 3 = -132? Wait, but we used a different loss? Actually, we used squared error, so derivative = -2*(8 - 10*3)*3 = -2*(8-30)*3 = -2*(-22)*3 = 132. But in our from_scratch, derivative = -2*(8-30)*3 = 132? Let me recompute: prediction = 10*3 = 30, loss = (8-30)^2 = 484, derivative = -2*(8-30)*3 = -2*(-22)*3 = 132. Yes.

# Update weight using gradient descent
learning_rate = 0.1
with torch.no_grad():
    weight -= learning_rate * weight.grad

# After update, weight is now 10.0 - 0.1*132 = 10.0 - 13.2 = -3.2? That's too big. But we want a small nudge, so learning rate should be 0.001. Anyway, the mechanism is the same.

print(weight)
```

Underneath, `loss.backward()` computes the derivative for every weight in the model — the same arithmetic we wrote by hand.


## What the code does

The plain Python code defines a weight, computes the derivative of the loss with respect to that weight for a single record, and then subtracts a small multiple (learning rate) of that derivative. This is the core of gradient-based approximate unlearning: you nudge the weights in the direction that reduces the influence of the record. The framework code (PyTorch) does the same thing but at scale: `loss.backward()` computes all gradients automatically, and you update the weights in a loop. The learning rate is crucial — too high and the model forgets too much, damaging other knowledge (utility); too low and the record's influence persists.


## Common misconceptions

**Approximate unlearning is just a quick fix and always works** — I can just run a few gradient steps on the record I want to forget, and it's gone.

A few steps might reduce the record's influence, but they can also damage the model's performance on other data, and they don't guarantee the record is fully forgotten. The influence can be spread across many weights, and a local nudge might miss distant ripples. In fact: Approximate unlearning is a heuristic that reduces influence, but it must be carefully tuned (learning rate, number of steps) and verified (e.g., using membership inference attacks) to ensure both forgetting and utility preservation.

**The learning rate is a minor detail** — The learning rate is just a number; any small value will do.

If the learning rate is too large, the weight update can overshoot, causing the model to forget not just the record but also valuable patterns (utility collapse). If it's too small, the record's influence remains, and the unlearning is ineffective. In fact: The learning rate is a critical hyperparameter that must balance between forgetting the target record and preserving the model's overall performance. It is batch-specific and often requires tuning per request.

**Bounded guarantee means the model is definitely unbiased** — Since I've set a bound on the influence, the model is now perfectly fair and unbiased.

A bounded guarantee on influence does not address fairness or bias; it only limits the effect of a single record. Other biases from the remaining data persist. In fact: Approximate unlearning reduces a specific record's influence, but it does not magically remove all biases. Fairness and bias are separate concerns that require their own evaluations and corrections.

**Approximate unlearning is the same as exact unlearning if you run it long enough** — If I just run approximate unlearning for many iterations, it will eventually match retraining.

Gradient-based nudges are local and cannot, in general, recover the exact retrained model. The model's parameters are a complex function of all data; removing one record changes the entire landscape, and a few local steps cannot undo the global effect. In fact: Approximate unlearning aims for a close approximation, but it will never be exactly the same as retraining from scratch. Exact unlearning requires structuring training in advance (e.g., SISA) to make exact isolation tractable.

**Verification of approximate unlearning is trivial** — I can just check the loss on the record; if it's high, the model forgot it.

A high loss on the deleted record does not guarantee the model won't leak information in another way (e.g., through memorised patterns that are still embedded in weights elsewhere). In fact: Verification requires a suite of tests, including membership inference attacks, to assess whether the record can still be distinguished from unseen data. A single loss value is insufficient.


## In short

Approximate unlearning is a fast, practical way to reduce a record's influence on a trained model by adjusting weights using gradients, without full retraining. It trades away the guarantee of exact removal. The key is balancing the learning rate to forget the target without degrading the model. It's used in LLMs, on-device models, RAG, agents, and everyday ML, but it must be verified and tuned carefully.


## Where this is used

Approximate unlearning is not just a theoretical curiosity — it's a practical tool in the AI engineer's toolkit, used when full retraining is infeasible. Here's where it actually shows up:

- **Large language models (LLMs)** — These models, such as GPT-4, have billions of parameters and are trained on massive corpora. If a user exercises their 'Right to Be Forgotten' and requests removal of their data, retraining the entire model is astronomically expensive (months of training on thousands of GPUs). Instead, approximate unlearning methods apply a gradient-based correction: they compute the derivative of the loss on the offending record and update the weights to 'forget' it. This happens per request, and multiple requests can be batched. The trade-off is that some residual influence may remain, which is why regulators are still evaluating whether such measures meet legal standards.
- **Small models on devices (TinyML)** — Think of a smart speaker that recognises your voice. If you ask to be forgotten, the device has a small model trained locally. Retraining from scratch on the device would drain the battery and require sending the data back to the cloud. Instead, local approximate unlearning tweaks the model's weights directly on the device, using the gradient computed from your voice samples. The assumption is that your vocal patterns only affect a small subset of weights, so a quick nudge suffices.
- **Retrieval-augmented generation (RAG)** — RAG systems combine a generative model with a retrieval index of documents. If a document must be removed (e.g., copyright violation), the index is updated (removing the entry), but the generative model may have been fine-tuned on it. Approximate unlearning can be applied to the model to reduce the influence of that specific document, preventing it from being regurgitated in responses. The mechanism: compute the gradient of the model's loss on the document's text and update the weights to reduce the likelihood of generating similar content.
- **Agentic systems that plan and call tools** — Imagine an AI agent that has been trained on user-specific data to help with calendar management. When a user deletes their account, the agent must forget their calendar patterns. Full retraining is expensive; instead, the agent applies approximate unlearning to its policy and value networks, nudging the weights away from the deleted user's patterns. This keeps the agent functional for other users while attempting to erase the specific individual's influence.
- **Everyday applied machine learning** — In recommendation systems (Netflix, Amazon), user behaviour data is used to train collaborative filtering models. When a user deletes their account, these systems need to forget their preferences. Approximate unlearning allows the recommendation model to be updated in near-real time, without the need to retrain the model over the entire user base. Similarly, in fraud detection, if a transaction is found to be misclassified, the model can be nudged to reduce its reliance on that pattern, improving accuracy without a full retrain.

In practice, a developer would implement approximate unlearning using a machine learning framework like PyTorch or TensorFlow. They would load the trained model, compute the loss on the specific records to forget, call `loss.backward()`, and then update the weights using an optimizer (e.g., SGD or Adam) with a carefully chosen learning rate and a few iterations. What breaks when done badly? If the learning rate is too high, the model degrades on unrelated data (utility loss); if it's too low, the record's influence persists (incomplete forgetting). The choice of which records to forget, and how to compute the gradient, is also critical — using the wrong loss function can inadvertently forget unrelated information.


---

Read this on the web: https://basavapurushottam.com/learn-ai/approximate-unlearning
