# Utility Preservation

Keeping a model useful after unlearning — the central tension between forgetting a record and degrading everything else.

**Leads to:** Approximate Unlearning, Exact Unlearning, Federated Unlearning

**Rests on nothing.** A place to start.

*Machine Learning · Machine Unlearning*


## Why learn this

This concept unlocks a deeper look at machine unlearning — the field of removing specific data from a trained model. Without understanding utility preservation, you cannot judge whether any unlearning method works: it is the yardstick by which every forgetting technique is measured. After this, you can understand Exact Unlearning, Approximate Unlearning, and SISA Sharded Training, and see why they are structured the way they are. It also prepares you for the legal and operational side — Deletion Requests and the Right to Be Forgotten — because both hinge on the same question: can we forget without breaking the model?


## The intuition

Imagine you have a shelf of recipe books. You made a note in one book with a friend's egg allergy, and now you must remove that note — privacy rules say so. If you rip out that one page, the book might lose a few recipes, but the rest of the book stays useful. But what if the note was on a page with four other recipes? You can't just tear it out; you'd lose all four. You could rewrite the page, but that changes the other recipes slightly. Now imagine the whole library: every book shares pages, and removing one note affects many books. The challenge is: how do you take out the one note without ruining the other recipes? That is the 'utility' — how useful the books remain for the other readers. If you get too aggressive and tear out too much, the library becomes useless for everyone. This is the core tension: forget the one thing, but keep everything else working.


## An analogy

Think of a chef who has trained for years by tasting thousands of dishes. A customer remembers they once watched the chef cook a secret family recipe and asks that the training be 'forgotten'. The chef must 'unlearn' that specific dish — but they cannot wipe their entire knowledge of cooking. They need to forget the one recipe while keeping all other skills: knife work, sauces, timing. If they 'forget' too aggressively, they might forget how to season properly or how to control heat. That's utility loss. But if they forget too little, the recipe lingers, and the request isn't honored. The chef's brain is a model; the recipe is a data point. The challenge is to erase one memory without erasing the surrounding skills. This analogy breaks down because a human brain isn't a mathematical model, but it captures the idea: unlearning is a targeted deletion, not a learning reset. In a real model, every input is interwoven with thousands of others in its weights — so removing one point is like editing a single thread in a tapestry; pull too hard and the whole picture distorts.


## Definition

Utility preservation is the property of an unlearning procedure that, after removing the influence of specific data, the model retains its ability to perform well on all other tasks and data.


## How it connects

You have not yet studied machine unlearning, but this concept is the first step. In your library, Machine Unlearning is the parent concept, and utility preservation is one of its key ideas. Later, you will see how Exact Unlearning achieves perfect utility by retraining from scratch, and how Approximate Unlearning trades utility for speed. This concept also connects to the Right to Be Forgotten, because the legal right to deletion only matters if the model remains useful afterward — otherwise, the whole model becomes worthless.


## The mathematics

This concept uses **statistics**, which is the mathematics of making decisions from data. In machine learning, we care about whether a model works well on new information it has never seen before. This is called **generalization**. To check if a model generalizes, we compare its performance before and after we remove some training data.

We measure performance using **accuracy**. Accuracy is the percentage of correct predictions out of all predictions made.

Here is how to calculate accuracy:
1. Count the total number of test cases (examples the model has not seen during training).
2. Count how many of those test cases the model predicted correctly.
3. Divide the number of correct predictions by the total number of test cases.
4. Multiply the result by 100 to get a percentage.

Formula for Accuracy:
Accuracy = (Number of Correct Predictions / Total Number of Test Cases) × 100%

**Utility Preservation** checks if removing data hurts this accuracy. We calculate it by comparing the accuracy after unlearning to the accuracy before unlearning.

Formula for Utility Preservation:
Utility Preservation = (Accuracy After Unlearning / Accuracy Before Unlearning) × 100%

If the result is close to 100%, the model kept its utility. If the result is much lower, the model lost significant ability to predict correctly on new data.

Note: We do not use complex algebra or calculus here. We only use division and multiplication of percentages.


## A worked example

We have a model that predicts whether a customer will leave a service (churn). It was trained on records from 10,000 customers. To test it fairly, we use a **test set** of 100 customers whose data was NOT used to train the model.

**Step 1: Calculate accuracy before unlearning.**
We ask the model to predict the churn status for all 100 customers in the test set.
The model gets 85 predictions correct.
Total test cases = 100.
Correct predictions = 85.
Accuracy Before Unlearning = (85 / 100) × 100% = 85%.

This 85% is our baseline. It tells us how useful the model is when it has all its training data.

**Step 2: Perform unlearning.**
We process a request to forget the records of 10 specific customers from the training set. The model is updated (or retrained) without these 10 records.

**Step 3: Calculate accuracy after unlearning.**
We test the same 100 customers in the test set again with the new, updated model.
The model now gets 70 predictions correct.
Total test cases = 100 (unchanged).
Correct predictions = 70.
Accuracy After Unlearning = (70 / 100) × 100% = 70%.

**Step 4: Calculate Utility Preservation.**
We compare the new accuracy to the old accuracy.
Utility Preservation = (70% / 85%) × 100%

First, divide 70 by 85:
70 ÷ 85 ≈ 0.8235

Then, multiply by 100:
0.8235 × 100 = 82.35%

**Result:**
The Utility Preservation is approximately 82.4%.

This means the model retained about 82.4% of its original predictive power after forgetting the data. The remaining 17.6% loss in utility indicates that removing those 10 records did harm the model's ability to generalize to new customers. In some applications, a drop this large might be considered unacceptable.


## From scratch, in Python

```python
# First, let's measure accuracy before unlearning.
# We'll simulate a simple model that makes predictions, and we have a list of true answers.

# A list of predicted labels (1 = churn, 0 = not churn)
predictions = [1, 0, 1, 1, 0, 1, 0, 0, 1, 0]
# The true labels for the same 10 test samples
true_labels = [1, 0, 0, 1, 0, 1, 0, 0, 1, 1]

# Count how many are correct
def accuracy(preds, trues):
    correct = 0
    for i in range(len(preds)):
        if preds[i] == trues[i]:
            correct = correct + 1
    return correct / len(preds)

acc_before = accuracy(predictions, true_labels)
print("Accuracy before unlearning:", acc_before)

# Now simulate unlearning by making some predictions worse (to represent collateral damage)
# After unlearning, the model gets 2 predictions wrong that it got right before
predictions_after = [1, 0, 0, 1, 0, 1, 0, 0, 1, 0]  # second and last changed

acc_after = accuracy(predictions_after, true_labels)
print("Accuracy after unlearning:", acc_after)

# Utility preservation ratio
utility_preserved = (acc_after / acc_before) * 100
print("Utility preserved: {:.1f}%".format(utility_preserved))
```


## The same thing, with a framework

```python
In practice, you would use a library like scikit-learn to compute accuracy easily, but the underlying math is the same. For example: from sklearn.metrics import accuracy_score; accuracy_score(true_labels, predictions). This does the same counting loop internally.
```


## What the code does

The code defines a function `accuracy` that compares two lists item by item. It uses a `for` loop to go through each index i, and counts how many times the prediction matches the true label. The `range(len(preds))` gives the numbers 0, 1, 2, ... up to the length of the list minus one. The function returns the fraction correct. We compute accuracy before and after a simulated unlearning (where two predictions are deliberately changed to be wrong). The utility preserved is simply the after accuracy divided by the before accuracy, multiplied by 100 to get a percentage. This shows the same arithmetic as the mathematics section: compare before and after to quantify the drop.


## Common misconceptions

**Unlearning is just deleting the data from the database.** — If I just remove the record from the training dataset, the model doesn't have it anymore, so it's forgotten.

The model was trained on the data, so its weights have been influenced by that record. Deleting the record from the database does not undo the influence on the weights. In fact: Unlearning means modifying the model's weights to remove the influence of the forgotten data, not just removing data from storage.

**Utility drop is inevitable, so we just accept any amount.** — There's no way to forget something without losing some accuracy, so any drop is fine.

While some drop is often unavoidable, a severe drop can make the model useless. The goal is to minimize the drop so the model remains viable for its intended purpose. In fact: Utility preservation is a design goal: we want to remove the targeted data with minimal impact on the model's overall performance, and we quantify that impact to decide if the method is acceptable.

**Utility only matters for the deleted records.** — As long as the model no longer remembers the deleted user, it's fine.

The model's utility is about its performance on all data, not just the deleted records. If the model forgets how to make general predictions, it is useless. In fact: Utility is measured on a held-out test set of unrelated data — the model must still perform well there.


## In short

Utility preservation is the delicate balance in machine unlearning: forget the targeted data, but keep the model useful for everything else. It is quantified by comparing the model's performance on a test set before and after unlearning. A large drop indicates collateral damage, which is unacceptable. The challenge is that a model's weights are interconnected, so removing one record's influence can degrade other capabilities. The goal is to minimize this degradation while ensuring the forgetfulness is effective.


## Where this is used

Utility preservation is the central concern in machine unlearning, which is used in several real-world applications:
* **Large language models (LLMs):** An LLM trained on web data might contain personal information about individuals. If someone requests deletion, the model must be updated to forget that data. The challenge is that the model's weights encode billions of parameters, and the specific data's influence is distributed across them. A simple fine-tuning step to 'forget' might unintentionally degrade the model's ability to write coherent text or answer general questions — a loss of utility. Developers address this by using approximate unlearning methods that adjust weights in a way that minimizes the impact on other capabilities. They also evaluate utility by testing the model on a diverse benchmark of tasks (like question answering, translation, and reasoning) before and after unlearning, ensuring the drop is below a certain threshold.
* **Small models on devices:** Small models, like those in a smartphone's keyboard or a voice assistant, are typically fine-tuned from a larger model. If a user asks to delete their typing history, the on-device model must be updated. Because these models are small, they may be more susceptible to utility loss — every parameter matters. Developers might use exact unlearning techniques like SISA (if the training was structured that way) or approximate methods with careful tuning. The evaluation is even more critical because the model's performance on other users' data must remain high.
* **Retrieval-augmented generation (RAG):** RAG systems retrieve relevant documents from a database and feed them into an LLM. Unlearning in RAG is often about removing documents from the database rather than changing the LLM's weights. In this case, utility preservation means ensuring that after removing a document, the retrieval still finds other relevant documents and the generation remains fluent. The 'model' here is the whole pipeline, and utility is measured by the quality of answers on a test set of queries.
* **Agentic systems:** Agents that plan and call tools might have been trained on trajectories that include user-specific data. If a user requests deletion, the agent's policy must be updated. Utility preservation here means the agent must still be able to plan and call tools correctly on general tasks. Since the agent's behavior is complex, evaluation is multi-faceted — test on a suite of planning benchmarks. The developer might use reinforcement learning to fine-tune the policy after unlearning, but must be careful not to over-correct.
* **Everyday applied ML:** In credit scoring, fraud detection, or recommendation systems, unlearning is used to comply with deletion requests. A bank might need to forget a customer's loan history. Utility preservation means the model's overall accuracy on other customers' creditworthiness should not degrade significantly. Developers would measure AUC (area under the ROC curve) or accuracy on a validation set before and after unlearning, and set a threshold for acceptable loss — say, at most 2% drop. If the drop exceeds that, they might fall back to full retraining, which is the gold standard for utility preservation.


---

Read this on the web: https://basavapurushottam.com/learn-ai/utility-preservation
