# Unlearning Verification

Demonstrating that a model has genuinely forgotten specific data rather than merely appearing to.

**Rests on:** Exact Unlearning

*Machine Learning · Model Evaluation*


## Why learn this

If you want a model to forget data a user asked to remove — their photo, their chat history, their financial record — you need to *prove* it's really gone, not just hope so. This concept gives you the audit toolkit: how to check a model has forgotten, and why 'it doesn't answer the question anymore' is not the same as 'it has forgotten'. Without verification, every unlearning claim is just a guess.


## The intuition

Imagine a student has been taught a fact in class. The teacher tells them to forget it. The student says 'okay'. How do you know they actually did? They might be hiding it, or they might have genuinely forgotten. You can ask them the fact directly — but if they refuse to answer, that doesn't prove they don't know it; maybe they just don't want to say. You can give them a test that indirectly reveals whether they know it, without asking directly. That's what verification does for AI models. When a model is trained on your data, it might memorize pieces of it. Unlearning is the process of removing that memory. Verification is the process of checking that the removal actually happened — and it's surprisingly hard. The model doesn't talk, so you can't ask 'did you forget?'. Instead, you use clever tricks to sniff out whether the memory is still there, hidden inside the model's billions of numbers. The intuition: *forgetting is invisible; you have to design a test to catch it*.


## An analogy

Think of a witness who has seen a criminal's face. The defence wants them to 'forget' it before testifying. You can't just ask 'do you remember the face?' because they could lie. You could show them a line-up of photos — if they flinch at one, you know they remember. But the witness could also be a skilled liar and control their flinch. So you make the test subtler: you measure their heart rate, their pupil dilation, their reaction time. Those are harder to fake. This is exactly what membership inference attacks do to models. They probe the model with carefully chosen inputs and measure its *confidence* — a model that remembers a piece of data tends to be overconfident about things related to it. If you can show that the model's confidence about the forgotten data is the same as it would be for data it has never seen, you have evidence it has forgotten. The analogy breaks down here: a witness has a single memory, but a model has billions.


## Definition

Unlearning verification is the process of testing a machine learning model to demonstrate — with evidence — that it has genuinely lost the influence of specific training data, rather than merely behaving as if it has forgotten. A membership inference attack is a method used in this process to check if specific data was in the training set by measuring whether the model's confidence — its internal certainty expressed as probability scores — is abnormally high for that data. These measurements rely on the model's weights, which are the internal settings that determine how the model processes information.


## How it connects

This concept sits inside Model Evaluation — the discipline of measuring whether a model works. Unlearning verification is a special kind of evaluation: instead of measuring accuracy on new data, we measure absence of memory on old data. It connects directly to Membership Inference (the attack used as the audit tool) and Right to Be Forgotten (the legal driver).


## The mathematics

The branch of mathematics this concept draws on is **probability** — the mathematics of uncertainty. Verification is all about measuring *how confident* a model is, and confidence is a probability. We need probability because we can't ever be 100% sure the model has forgotten; we can only get strong *statistical evidence*. The operation we actually use is called a **hypothesis test**. Here's how it works in plain words.

We want to know: 'Does the model behave like it has never seen this data?' We can't check its memory directly, so we ask an indirect question: 'When we probe the model with inputs that are related to the forgotten data, is its confidence *abnormally high*?' If a model has memorized a piece of data, it tends to be overly sure about it. We compare the model's confidence on the forgotten data to its confidence on a bunch of *similar but never-seen* data. If the forgotten data's confidence is within the normal range of the never-seen data, we say: 'We cannot find evidence that it remembers.' This is a probability statement, not a proof.

To make this comparison precise, we use a number called the **z-score**. The z-score tells us how many standard deviations away from the average our specific score is. A standard deviation is a measure of how spread out the scores are. If the z-score is large (usually greater than 2 or less than -2), it means the score is far from the typical range, which suggests something unusual is happening.

Here is the step-by-step calculation:

1. **Calculate the average confidence** of the control group (the data the model never saw). Let's call this `mean_control`.
2. **Calculate the standard deviation** of the control group. This tells us how much the control scores typically vary from their average. Let's call this `std_control`.
3. **Get the confidence score** for the specific data point we are testing (the one we suspect was memorized). Let's call this `score_test`.
4. **Compute the z-score** using this formula:

   z = (score_test - mean_control) / std_control

   This formula subtracts the average from our test score to see how far it is, then divides by the standard deviation to scale that distance relative to the typical variation.

5. **Check the threshold**. If the absolute value of z (ignoring whether it is positive or negative) is greater than 2, we flag the data point as a probable member of the training set. This means its confidence was unusually high compared to the never-seen data.


## A worked example

```text
We have a model that was trained on a dataset including one user's profile. The user requested deletion, and the company claims the model has forgotten them. We need to verify. We have 10 other profiles that were *never* in the training data (call them 'control' profiles). We ask the model to predict the likelihood that each profile is a customer (a number from 0 to 1).

Step 1: Control profiles' confidence scores: [0.12, 0.15, 0.09, 0.13, 0.11, 0.14, 0.10, 0.12, 0.13, 0.11]   # These are the model's confidence that each control profile is a customer. They are all around 0.10-0.15, because the model has never seen them and is not sure.

Step 2: Compute the average of control scores (mean_control):
   Sum = 0.12 + 0.15 + 0.09 + 0.13 + 0.11 + 0.14 + 0.10 + 0.12 + 0.13 + 0.11 = 1.20
   mean_control = 1.20 / 10 = 0.12
   This is the baseline 'normal' confidence for never-seen data.

Step 3: Compute the standard deviation (std_control):
   First, find how far each score is from the average (0.12), square that distance, and sum them up:
   (0.12-0.12)^2 = 0
   (0.15-0.12)^2 = 0.03^2 = 0.0009
   (0.09-0.12)^2 = -0.03^2 = 0.0009
   (0.13-0.12)^2 = 0.01^2 = 0.0001
   (0.11-0.12)^2 = -0.01^2 = 0.0001
   (0.14-0.12)^2 = 0.02^2 = 0.0004
   (0.10-0.12)^2 = -0.02^2 = 0.0004
   (0.12-0.12)^2 = 0
   (0.13-0.12)^2 = 0.01^2 = 0.0001
   (0.11-0.12)^2 = -0.01^2 = 0.0001
   Sum of squared differences = 0.0030
   
   Variance = Sum / N = 0.0030 / 10 = 0.0003
   Standard deviation (std_control) = square root of variance = square root of 0.0003 ≈ 0.0173
   We will use 0.0173 for the calculation.

Step 4: Now ask the model for the deleted user's profile. Its confidence score (score_test): 0.45   # This is much higher than the average of 0.12.

Step 5: Compute the z-score:
   Difference = score_test - mean_control = 0.45 - 0.12 = 0.33
   z = Difference / std_control = 0.33 / 0.0173 ≈ 19.07

Step 6: Check the threshold.
   Is |19.07| > 2? Yes, 19.07 is much larger than 2.

Result: The model's confidence on the deleted user's profile is far outside the range of never-seen data (more than 19 standard deviations away). This is strong evidence that the model *still remembers* the deleted user. The verification fails.
```


## From scratch, in Python

```python
# We'll simulate a model's confidence scores.
# In reality, you'd call your model's predict_proba() function.

# Step 1: Collect confidence scores for control data (known non-members)
control_scores = [0.12, 0.15, 0.09, 0.13, 0.11, 0.14, 0.10, 0.12, 0.13, 0.11]

# Step 2: Calculate mean (average)
def mean(scores):
    return sum(scores) / len(scores)

mean_control = mean(control_scores)
print("Mean control score:", mean_control)

# Step 3: Calculate standard deviation (spread)
def std(scores):
    m = mean(scores)
    variance = sum((s - m) ** 2 for s in scores) / len(scores)
    return variance ** 0.5

std_control = std(control_scores)
print("Standard deviation:", std_control)

# Step 4: Get the model's confidence on the deleted user's data
# (In practice, you'd call your_model.predict_proba(deleted_user_data)[1])
deleted_user_score = 0.45

# Step 5: Compute the z-score
z_score = (deleted_user_score - mean_control) / std_control
print("Z-score:", z_score)

# Step 6: Set a threshold (commonly 2)
threshold = 2
if abs(z_score) > threshold:
    print("Likely still remembers the deleted user. Verification FAILED.")
else:
    print("No evidence of memory. Verification PASSED.")
```


## The same thing, with a framework

```python
import numpy as np

control_scores = np.array([0.12, 0.15, 0.09, 0.13, 0.11, 0.14, 0.10, 0.12, 0.13, 0.11])
mean_control = np.mean(control_scores)
std_control = np.std(control_scores)

# For the deleted user, get the model's confidence
# (e.g., from a PyTorch model: torch.sigmoid(model(deleted_user_input)).item())
deleted_user_score = 0.45

z_score = (deleted_user_score - mean_control) / std_control
flag = abs(z_score) > 2
print(f"Z-score: {z_score:.2f}, Flagged: {flag}")
```


## What the code does

The from-scratch code does the following: first, we store a list of ten control scores — these are the model's confidence values on data we know it has never seen. The `mean` function sums all the numbers and divides by how many there are, giving the average. The `std` function measures how spread out the numbers are — it computes the average of the squared differences from the mean, then takes the square root. Then we have the deleted user's score, which we plug in. The z-score is just how many standard deviations the deleted user's score is from the control mean. If it's more than 2 standard deviations away, we suspect the model remembers. The framework version uses NumPy's built-in `mean` and `std`, which do exactly the same arithmetic under the hood. The key insight: the control scores give us the 'normal' range, and the z-score tells us if the deleted data is an outlier.


## Common misconceptions

**If the model doesn't output the deleted text, it's forgotten.** — I asked the model about the user's data and it said it didn't know. So it must have forgotten.

The model might still know it but be trained to refuse to reveal it. Refusal is not the same as ignorance. In fact: Verification must use indirect methods like membership inference — probing the model's confidence on related inputs, not just asking directly.

**Retraining from scratch is always the best verification.** — The only true verification is to retrain the model without the data and compare.

Retraining is expensive and often impossible for large models. It's the gold standard for small models, but verification methods exist for approximate unlearning. In fact: For small models, retraining is a good baseline. For anything large, you need statistical tests like the z-score.

**A high confidence on the deleted data means the model remembers.** — I saw the model was 90% confident on the deleted user's data, so that proves it remembers.

Confidence is relative. The model might be 90% confident on everything, or the deleted data might be similar to other data. You need to compare to control data. In fact: The z-score compares the deleted data's confidence to the distribution of confidences on never-seen data. Only if it's an outlier is it suspicious.

**Verification is a one-time check.** — Once I verify that the model has forgotten, I can be sure forever.

Models can be updated, attacked, or fine-tuned later, potentially bringing back the memory. In fact: Verification should be repeated after any model update or after a significant time has passed.


## In short

Unlearning verification is the process of gathering evidence that a model has truly forgotten data. Because you can't look inside the model, you use probabilistic tests — membership inference — that compare the model's confidence on the forgotten data to its confidence on never-seen data. If the forgotten data behaves like never-seen data, you have evidence of forgetting. If it stands out, you don't. The z-score test is the basic tool: it tells you how unusual the forgotten data's confidence is. This evidence is what makes 'unlearning' a real claim rather than a guess.


## Where this is used

Unlearning verification is the quality gate for any system that must comply with deletion requests. Here's how it's used in practice:

- **Large Language Models (LLMs):** LLMs are trained on billions of documents scraped from the internet, including personal data. When a user asks to be forgotten, the provider claims to 'unlearn' that data. Verification is done by running membership inference attacks: generate prompts that would uniquely identify the user (e.g., their full name and a phrase from their blog) and see if the model completes them verbatim. If the model outputs the exact text, it's a red flag. Developers write scripts that sample thousands of such prompts and measure the model's perplexity (a measure of surprise) — if the model is suspiciously fluent on the deleted text, it has likely memorized it. The challenge is that you can't easily roll back an LLM's training, so many providers use 'approximate unlearning' (fine-tuning the model to forget) — and verification is the only way to know if that actually worked. The developer's workflow: after applying an unlearning method, run a suite of MIA tests on a held-out set of known personal data, and set a threshold. If the attack success rate is above a baseline (compared to control data), the unlearning failed and you have to try a stronger method.

- **Small models on devices (on-device unlearning):** On a phone, a model might be fine-tuned for a user's voice or typing style. When they wipe the device, the model must forget that user. Verification here is simpler because the model is small — you can run exact unlearning (retran on the data minus the user's) on the fly and compare the two models. The verification step is: retrain a baseline model on the same data *without* the deleted user, and then compare the production model's outputs to the baseline's on the user's data. If they are statistically indistinguishable, unlearning worked. This is a strong form of verification but it is expensive — that's why SISA-style sharding exists, to make retraining fast. On a phone, you have limited compute, so you might instead use a lightweight membership inference test: measure the model's confidence on the user's data and compare to control data. If the confidence is not abnormally high, you approve.

- **Retrieval-Augmented Generation (RAG):** In RAG, the model doesn't store your data — it retrieves from a vector database. So the 'unlearning' is trivially done by deleting the vectors from the database. But verification is still crucial: you must confirm the vectors are actually gone, and that the model no longer generates that information from its parametric memory (because the base model might have memorized it during pre-training). The verification procedure: (1) check the vector database is empty for that user's items (a simple query); (2) ask the model, without any retrieval, to generate text about the user — if it still produces the data, you have a problem. So verification is a two-part check: external storage and internal memory.

- **Agentic systems (AI agents that plan and call tools):** An agent might have access to a tool that searches a database containing personal records. If a user's data is deleted from the database, the agent should no longer be able to retrieve it. But the agent might have memorized the data in its context or memory files. Verification for agents is about black-box testing: simulate a conversation where the user asks for the deleted data, and see if the agent either retrieves it from memory or calls a tool and gets an empty result. If the agent tries to call the tool and receives 'not found', that's a good sign. But if the agent inaccurately produces the data from its own weights, that's a failure. Developers write test scenarios that replay past user interactions, mask them out, and verify the agent's responses no longer contain the exact personal details.

- **Everyday applied ML (recommendation, vision, fraud):** In recommendation systems, user data like watched movies or past purchases is used to train collaborative filtering models. If a user deletes their account, the recommendation model must forget their preferences — otherwise, other users might see recommendations based on that person's history. Verification uses membership inference: generate synthetic users and see if the model's recommendations are unusually tailored to the deleted user's profile. In vision, if a model was trained on photos of individuals, unlearning verification would check whether the model can still recognize their face — by querying it on a cropped version of the photo and seeing if it produces a high-confidence match. In fraud detection, unlearning a particular transaction means the model should no longer flag similar transactions as part of that specific fraud ring. Verification here means checking that the model's false-positive rate on that ring's patterns drops back to the baseline. In all cases, the developer writes an evaluation script that compares the model's behavior on the 'forgotten' data to a control group, computes a statistical test (like the z-score), and gates deployment on the result.

In summary, verification is not an afterthought — it is the step that makes unlearning legally and technically meaningful. Without it, 'we unlearned your data' is just a promise. With it, you have evidence.


---

Read this on the web: https://basavapurushottam.com/learn-ai/unlearning-verification
