# SISA Sharded Training

Sharded, isolated, sliced and aggregated training — partitioning data so each record touches only one shard's model.

**Rests on:** Stochastic Gradient Descent

*Machine Learning · Machine Unlearning*


## Why learn this

This concept unlocks understanding of how modern AI systems can delete your data from a trained model — a legal requirement under the Right to Be Forgotten. By learning this approach, you'll be able to grasp the rest of the Machine Unlearning map: Exact Unlearning (full retraining is the gold standard but expensive), Approximate Unlearning (fast but not guaranteed), Federated Unlearning (the distributed version), and how to verify that deletion worked (Unlearning Verification). You'll also understand the central tension of Utility Preservation — how to forget without wrecking the model. This is the foundation for building responsible AI systems that comply with data protection regulations.


## The intuition

Imagine you've baked a cake. If someone tells you there was a bad egg in one batch of ingredients, you can't just pick the egg out — you'd have to bake a whole new cake without it. That's the problem with retraining a machine learning model from scratch: it's expensive and slow. This method solves this by baking many small cakes instead of one big one. You split your data into separate groups, train a separate model on each group, and then combine their votes to get a final answer. When a deletion request comes, you only need to rebake the one cake that used the bad egg — the one group's model that saw that record. The other groups are untouched. The catch is that each small cake is a bit more crumbly — the model trained on a smaller portion of data might be slightly less accurate than one trained on everything. That's the trade-off you accept for being able to delete cheaply.


## An analogy

Think of a committee making decisions. Instead of one all-knowing oracle, you have several experts, each trained on a different slice of the evidence. When a new case comes, each expert gives their opinion, and the committee votes (combines them). If one expert was given a piece of evidence that later must be forgotten, you don't need to retrain the whole committee — you just replace that one expert with a fresh one trained on the remaining evidence. The other experts are unaffected because they never saw the deleted evidence. This works because each piece of evidence is assigned to exactly one expert (divided into separate, isolated parts). But here's where the analogy breaks: a committee of experts is usually better than one expert who saw everything, but in machine learning, training on less data often makes each model worse. A model trained on 10,000 records is usually more accurate than one trained on 2,000. So this method sacrifices a little accuracy to gain the ability to delete efficiently. Also, the voting isn't just majority — it's often averaging the outputs, so if one model gives a wild answer, it's averaged out, but not if it's a critical part of the decision.


## Definition

This training method involves splitting data into separate parts, training independent models on each part, and combining their outputs for final predictions, such that deleting a record only requires retraining the single part that contained it.


## How it connects

This concept builds on nothing yet, but it connects to several neighbouring topics you'll encounter: Exact Unlearning (the goal of perfect removal — this method makes this affordable), Approximate Unlearning (a faster but weaker alternative), Federated Unlearning (similar idea but with updates instead of raw data), Deletion Requests (the operational process), Right to Be Forgotten (the legal driver), and Utility Preservation (the trade-off this method introduces). Your library notes, from the book *Machine Unlearning: Concepts, Techniques and Applications*, stress that this method makes exact unlearning tractable by bounding how much of the model any one record touched.


## The mathematics

This concept uses only two basic arithmetic operations: splitting a group into equal parts, and finding the average of several numbers. You do not need algebra or advanced statistics to understand how the system works.

First, consider how the data is organized. Imagine you have a list of records. To create 'shards', you divide this list into separate groups. For example, if you have 100 records and choose 4 shards, you split the list into 4 equal parts, each containing 25 records. The rule is strict: every record goes into exactly one shard. A record never appears in two different shards at the same time.

Next, consider how the system makes a prediction. Each shard trains its own separate model using only the records in that shard's group. When you want to predict something for a new case, each of these models gives an answer. Let us call these answers 'scores'. If the task is to estimate a probability, these scores are numbers between 0 and 1.

To get the final result, you combine these scores by taking their average. This means you add all the scores together and then divide by the number of models that gave them. This process is called aggregation.

Why does this make deletion easy? Because each record lives in only one shard, changing or removing that record only affects the model trained on that specific shard. The other models never saw that record, so they do not need to be changed. You only retrain the one affected shard and then recalculate the average.

Here is the step-by-step logic for the aggregation:
1. Let S be the total number of shards (groups).
2. For each shard i (where i is a number from 1 to S), let y_i be the score produced by that shard's model.
3. The final prediction, which we will call ŷ (read as 'y-hat'), is calculated by adding all the scores together and dividing by S.

In words: ŷ equals the sum of all individual scores divided by the number of shards.

This formula works because the average treats every shard's opinion equally. If you remove a record from shard j, you only need to update y_j. The other scores (y_1, y_2, etc.) remain exactly the same because their models were not touched.


## A worked example

Let us walk through a concrete example with real numbers.

Imagine we have a dataset of 6 loan applications. Each application has an income amount and a label: 1 means approved, 0 means rejected. We decide to split this data into 3 shards. Since there are 6 records, each shard will contain exactly 2 records.

Step 1: Training the models
We train three separate simple models. Model 1 uses the 2 records in Shard 1. Model 2 uses the 2 records in Shard 2. Model 3 uses the 2 records in Shard 3.

Step 2: Getting individual scores
Now we test a new applicant on all three models to see what they predict. We assume each model outputs a 'repayment score' between 0 and 1, representing confidence that the loan will be repaid.
- Model 1 (Shard 1) outputs a score of 0.7.
- Model 2 (Shard 2) outputs a score of 0.4.
- Model 3 (Shard 3) outputs a score of 0.2.

Step 3: Calculating the final prediction
We need to find the average of these three scores.
First, we add them together:
0.7 + 0.4 + 0.2 = 1.3

Next, we divide this sum by the number of shards, which is 3:
1.3 divided by 3 equals approximately 0.433.

So, the final prediction ŷ is 0.433. If our rule is to approve loans only if the score is 0.5 or higher, we would reject this loan because 0.433 is less than 0.5.

Step 4: The deletion scenario
Suppose we need to delete one record that was used in Shard 2. We do not touch Shard 1 or Shard 3. Their models and their scores (0.7 and 0.2) stay exactly the same.

We retrain only Model 2 using the remaining record in Shard 2. Let us say this new model now outputs a score of 0.5.

We recalculate the average with the new score:
Add the scores: 0.7 (from Shard 1) + 0.5 (new score from Shard 2) + 0.2 (from Shard 3) = 1.4.
Divide by the number of shards: 1.4 divided by 3 equals approximately 0.467.

The new final prediction is 0.467. The decision might change slightly, but we achieved this by retraining only one model instead of all three. This is the efficiency gain provided by the sharding structure.


## From scratch, in Python

```python
# Plain Python code to demonstrate SISA sharding and aggregation.
# We simulate training a 'model' as a simple average of the labels in each shard.

# Our dataset: list of (features, label) tuples. Features are ignored for simplicity.
data = [
    (10, 1),  # high income, approved
    (5, 0),   # low income, rejected
    (8, 1),   # medium-high, approved
    (3, 0),   # low, rejected
    (12, 1),  # very high, approved
    (2, 0)    # very low, rejected
]

# Step 1: Split into shards (e.g., 3 shards, 2 records each)
shard_size = 2
shards = []
for i in range(0, len(data), shard_size):
    shard = data[i:i+shard_size]  # slice the list
    shards.append(shard)

print("Shards:", shards)

# Step 2: Train each shard's model. Here 'training' means computing the average label.
def train_model(shard):
    labels = [label for (features, label) in shard]
    return sum(labels) / len(labels)  # average

models = [train_model(shard) for shard in shards]
print("Models (average labels):", models)

# Step 3: Aggregate predictions (average of models)
def aggregate(models):
    return sum(models) / len(models)

prediction = aggregate(models)
print("Aggregated prediction:", prediction)

# Step 4: Simulate a deletion request for the record (8, 1) which is in shard 2 (index 1).
# We need to remove that record from its shard.
deleted_features = 8

def delete_from_shard(shard, features):
    # Return a new shard without the record with those features
    return [record for record in shard if record[0] != features]

for idx, shard in enumerate(shards):
    if any(record[0] == deleted_features for record in shard):
        # Retrain only this shard
        new_shard = delete_from_shard(shard, deleted_features)
        print(f"Retraining shard {idx} after deletion, new shard: {new_shard}")
        models[idx] = train_model(new_shard)
        break

print("Updated models:", models)
print("New aggregated prediction:", aggregate(models))
```


## What the code does

- We create a list `data` with tuples; each tuple has features (income) and label (1/0). We ignore the features for simplicity.
- `shards` is a list of lists. The `range(0, len(data), shard_size)` gives starting indices 0, 2, 4, and we slice `data[i:i+shard_size]` to get 2 records each.
- `train_model` computes the average label in a shard, which simulates a trained model.
- `models` is a list of averages, one per shard.
- `aggregate` averages the model outputs, giving the final prediction.
- On deletion, we find the shard that contains the record with features=8. We rebuild that shard without that record and retrain only that shard's model. Other shards remain unchanged. Then we re-aggregate.
- This shows the key idea: only one shard's model changes, and the final prediction adjusts cheaply.


## Common misconceptions

**SISA requires no retraining at all** — With SISA, you never have to retrain; you just remove the data and the model is updated automatically.

SISA still requires retraining, but only on the affected shard. The other shards are untouched. The savings come from not having to retrain the entire model, which could be huge. In fact: SISA bounds retraining to a single shard, not a single record. You still retrain that shard's model from scratch on its remaining data.

**SISA guarantees no information leaks** — If I delete a record from a shard, the model that used it is gone, so no information about that record remains.

The other shard models might still have indirect correlations with the deleted record. Also, the aggregation step could leak some information if the other models were trained on overlapping features. SISA only ensures the direct influence of that record is removed; it doesn't guarantee zero leakage from correlations. In fact: SISA gives exact deletion in the sense that the record's direct influence is gone, but it doesn't address indirect leakage through correlated data in other shards.

**More shards always makes deletion cheaper** — If I increase the number of shards, deletion is cheaper because each shard is smaller, so retraining is faster.

While each retraining is faster, you also have more models to maintain and aggregate. The overhead of managing many small models can outweigh the savings. Also, each model sees less data, degrading accuracy (utility loss). There's a sweet spot. In fact: Increasing shards reduces per-shard retraining cost but increases storage/aggregation overhead and accuracy loss. You need a balance.

**SISA is always better than full retraining** — Why would anyone do full retraining when SISA is faster? SISA is the best method.

Full retraining gives the best possible utility because the model sees all the data. SISA trades some utility for deletion efficiency. In some high-stakes applications, the accuracy loss is unacceptable, so full retraining is still used despite the cost. In fact: SISA is a trade-off: you sacrifice a bit of accuracy to make deletion cheap. Full retraining remains the gold standard for model quality.

**SISA only works for tabular data** — SISA is a technique for databases, not for deep learning.

SISA is model-agnostic. You can shard any type of data — images, text, or tabular — as long as you can train separate models and aggregate their outputs. It's used in research with neural networks. In fact: SISA is a training strategy that can be applied to any machine learning model, including deep learning, by treating each shard as an independent training set.


## In short

SISA is a training strategy that makes exact unlearning tractable by sharding data into isolated parts. Each record touches only one shard's model, so on a deletion request you only retrain that shard. The final prediction is an aggregate of all shard models, typically the average. The cost is some utility loss because each model trains on less data. SISA underlies real-world systems that need to comply with data deletion laws, and it is a key building block for the entire field of machine unlearning.


## Where this is used

SISA is used in production systems that must comply with the Right to Be Forgotten. In practice, a developer would use a framework like PyTorch or TensorFlow to train each shard's model, but the core ideas map directly to the plain Python above.

- **Large language models (LLMs):** Training an LLM like GPT-4 on shards is impractical because the data is massive, but SISA-style principles appear in research on model unlearning for LLMs. The developer would partition the training corpus into shards, train separate smaller models, and aggregate their outputs (e.g., by averaging logits) to get a final response. When a deletion request arrives, only the shard containing that text is retrained. This is expensive but offers a strong guarantee.
- **Small models on devices:** For on-device keyword spotting (like 'Hey Siri'), SISA is more feasible due to smaller data. A developer using TensorFlow Lite would train a small model per shard (e.g., per user group), and aggregate by averaging outputs. A deletion request for a user only requires retraining that user's shard model and updating the on-device aggregate.
- **Retrieval-augmented generation (RAG):** RAG combines a retrieval system with a generative model. SISA can be applied to the retrieval index: instead of one huge vector database, you have several sharded indexes. To delete a document, you only remove it from its shard's index and rebuild that shard (e.g., recompute embeddings for that shard only). The aggregation happens at query time, where results from all shards are merged. This is a natural fit because the index is already sharded for scalability.
- **Agentic systems:** Agents that plan and call tools often use a memory of past interactions. If that memory is stored in a model trained on user data, SISA helps: each user's data goes to a separate shard model. When a user requests deletion, only that shard's model is retrained and the agent's behaviour changes. A developer building an agent with LangChain would have a per-user memory model and a global aggregation layer.
- **Everyday applied ML (recommendation, vision, forecasting, fraud):** In recommendation systems, SISA allows removing a user's interaction history from the model without retraining on all users. A developer using PyTorch would split user-item interactions into shards (e.g., by user ID), train a separate collaborative filtering model per shard, and aggregate scores (e.g., by averaging the predicted ratings). For fraud detection, sharding by account region allows deleting a region's data after a data breach. The cost is a slight drop in recommendation accuracy because each model sees less data.

**What a developer actually writes:** They define the sharding logic, e.g., `shard_id = hash(user_id) % num_shards`, and train each shard with the same network architecture. They store the shard models separately (e.g., in a directory per shard). On deletion, they receive a request, identify the shard(s), retrain those from the remaining data, and update the deployed models. Aggregation is often just averaging the outputs, but for classification it might be majority voting. The main thing that breaks when done badly is utility: if you shard too aggressively, each model underfits, and the aggregate accuracy drops. Also, if you don't ensure each record touches only one shard, the deletion guarantee is void.


---

Read this on the web: https://basavapurushottam.com/learn-ai/sisa-sharded-training
