# Federated Learning

Training a shared model across many devices or institutions that exchange updates rather than raw data.

**Rests on:** Pre-Trained Model Adaptation

*Security and Privacy · Privacy Engineering*


## Why learn this

Federated learning is the technique that lets an AI system improve using data that never leaves people's phones or a hospital's servers. Understanding it unlocks three things: (1) you will be able to read and judge privacy claims in AI products — does this really keep data local, or is it just marketing? (2) you will understand why some apps can improve their predictions without ever uploading your personal messages or health records; (3) you will be ready for the next steps in privacy engineering — differential privacy, which adds mathematical noise to protect individual data points, and secure aggregation, which lets multiple parties compute a shared result without seeing each other's inputs. These build directly on federated learning's core insight: share the update, not the data.


## The intuition

Think of a teacher who wants to improve a class assignment based on what students learned, but the students are not allowed to show their notebooks. Instead, each student does a small practice problem, then tells the teacher only one number: how much their answer would change if the lesson were adjusted. The teacher collects all these 'suggestions', combines them into a single adjustment, and updates the lesson for everyone. The notebooks never leave the students' desks. That is federated learning in essence: many devices (the students) each have private data (their notebooks). Instead of sending the data to a central server, each device trains a small copy of a shared model on its own data, then sends only the resulting update — a list of numbers showing how the model should change. The server averages these updates and improves the shared model. The raw data stays where it was born.

The key word is 'update'. An update is not the data itself — it is a change, a delta, like 'increase the weight for income by 0.2' or 'decrease the weight for debt by 0.1'. The server never sees the actual income or debt values; it only sees how the model should move. Over many rounds, the shared model becomes good at the task for everyone, even though no single person's private information was ever centralised.


## An analogy

A better analogy: a cooking class where the instructor wants to perfect a recipe, but each student has their own secret ingredient that they won't reveal. The instructor sends each student the current recipe. Each student cooks the dish at home, using their secret ingredient, and then reports back only one thing: 'If you increase the salt by a quarter teaspoon, the dish improves for me' or 'Decrease the pepper'. The instructor collects these suggestions from all students, averages them, and updates the recipe. After a few rounds, the recipe is better for everyone — but no student ever revealed their secret ingredient. This analogy holds well: the recipe is the model, the secret ingredient is the private data, the suggestion is the update, and the averaging is the server's job.

Where the analogy stops working: the students are giving *directions* (increase or decrease), but in real federated learning, the updates are precise numerical vectors — not just 'more salt' but 'change this specific weight by 0.03 and that one by −0.01'. Also, the instructor is not just averaging suggestions; they are running an optimisation algorithm to minimise a loss function. And crucially, in real federated learning, the data is not secret from the device itself — the device's own copy of the model can overfit to its local data if not handled carefully. The analogy also hides the problem of communication cost: in the cooking class, sending one number per student is cheap, but in federated learning, sending a full model update (which can be millions of numbers) is expensive — that is a central practical constraint.


## Definition

Federated learning is a machine learning technique in which a shared model is trained across many participating devices or institutions that each keep their own local data, and only exchange model updates (not raw data) with a central coordinating server.


## How it connects

You are learning this as a foundation in privacy engineering. Federated learning is one of the four key techniques you'll study here — alongside data minimisation (not collecting data in the first place), anonymisation (removing identifiers), and differential privacy (adding noise for formal guarantees). Federated learning is different from all of those because it changes *where* training happens, not *what* data is collected. It does not guarantee privacy by itself — as your library notes, updates can still leak information, which is why differential privacy is often combined with it. You haven't yet studied neural networks, but federated learning sits on top of standard gradient-descent training; you'll meet that later. For now, think of the 'model' as a black box that makes predictions, and the 'update' as a set of numbers that improve it. This concept builds on the idea of distributed systems — many computers working together — but you don't need any prior knowledge beyond that.


## The mathematics

Federated learning is not a new mathematical operation; it is a way of organising computation to protect data. But to understand the 'updates' that devices exchange, you must first understand what a model update is in plain words. This concept draws on **arithmetic** (averaging numbers) and on a branch called **calculus**, though you won't need to do calculus yourself — only one idea from it: a *gradient*, which tells you in which direction to change a model to make it better.

**The averaging step.** The server's job each round is to take the updates from many devices and combine them into one. If device A says 'change the first weight by +0.2', device B says 'change it by −0.1', and device C says 'change it by +0.3', then the average change is (+0.2 − 0.1 + 0.3) / 3 = +0.133. The server adds this average to the shared model's weight. That is the arithmetic of federated averaging — nothing more complicated than means.

**The gradient.** To ask the question 'how should the model change?', you need to know how the model's error changes when you tweak each weight. That rate of change is the gradient. It is a list of numbers, one for each weight, where each number says 'if this weight increased by 1, the error would increase by this much' (if positive) or 'decrease by this much' (if negative). If the gradient for a weight is −0.5, that means increasing that weight by 1 would *reduce* the error by 0.5 — so you want to move against the gradient, increasing the weight when the gradient is negative, to lower the error. In federated learning, each device computes its own gradient based on its own local data. That gradient is the 'update' sent to the server.

For round t, with N participating devices, each having a local model update Δw_i (a vector of gradient-based changes):

w_{t+1} = w_t + (learning_rate / N) × Σ_{i=1}^{N} Δw_i

where:
- w_t is the shared model's parameter vector at round t
- Δw_i is the update (negative gradient) computed by device i on its local data
- learning_rate (η) controls the step size
- N is the number of devices in this round
- Σ is the summation symbol, meaning 'add up all the Δw_i'


## A worked example

```
Imagine a very simple model with just one weight, w, and a constant error function. We have three devices, each with its own local data. Their local gradients for w are: Device A: −0.2 (error goes down if w increases), Device B: +0.1 (error goes up if w increases), Device C: −0.3 (error goes down if w increases). The server's current weight is w = 2.0. We'll use a learning rate of 0.1.

Step 1: Average gradient = (−0.2 + 0.1 + (−0.3)) / 3 = −0.4 / 3 = −0.1333   # This is the arithmetic mean of the three local gradients.
Step 2: Learning rate = 0.1, so update step = 0.1 × (−0.1333) = −0.01333   # The learning rate scales how big a step we take in the direction of improvement.
Step 3: New weight = old weight − update step = 2.0 − (−0.01333) = 2.01333   # We move the weight opposite to the gradient (because a negative gradient means we should increase the weight to reduce error). Here the gradient is negative, so we move the weight up.
Step 4: Rounds repeated: after 10 rounds, weight might become 2.1, 2.2, ...   # Each round repeats the same process with fresh local gradients, gradually improving the shared model.

Result: After one round of federated averaging, the shared weight is updated from 2.0 to 2.01333, moving in the direction that reduces average error across devices.
```


## From scratch, in Python

```python
# Federated learning with simple averaging
# No frameworks — just plain Python lists and loops.

# --- Building blocks ---
# A list is an ordered collection, e.g. [1, 2, 3].
# A dictionary maps keys to values, e.g. {'device1': [0.1, -0.2]}.
# A for loop repeats a block for each item in a list.

# Step 1: Each device has local data (but in reality, it would train a model)
# Here we simulate that each device returns a gradient vector (a list of numbers).
# The gradient is the 'update' that would be sent to the server.

# Define local updates as a dictionary: device id -> list of gradient values
device_updates = {
    'device_A': [0.2, -0.1, 0.3],   # three weights, for example
    'device_B': [-0.1, 0.4, 0.1],
    'device_C': [0.0, -0.2, 0.2]
}

# Step 2: The server has a current model (a list of weights)
# Initially, let's set all weights to 1.0
current_weights = [1.0, 1.0, 1.0]

# Step 3: Average the gradients across devices, element-wise
# First, count how many devices we have
num_devices = len(device_updates)   # gives 3

# We'll build a list of averages, one for each weight index
average_gradients = []
num_weights = len(current_weights)  # gives 3

for i in range(num_weights):        # i goes 0, 1, 2
    total = 0.0
    for device_id, grad_list in device_updates.items():
        total += grad_list[i]       # add each device's i-th gradient value
    avg = total / num_devices
    average_gradients.append(avg)   # add to the list

# Step 4: Update the model using a learning rate
learning_rate = 0.1
new_weights = []
for i in range(num_weights):
    # Gradient descent: move opposite to the gradient
    new_w = current_weights[i] - learning_rate * average_gradients[i]
    new_weights.append(new_w)

# Example output for this small case:
# average_gradients = [0.0333, 0.0333, 0.2]
# new_weights = [0.9967, 0.9967, 0.98]

print("Average gradients:", average_gradients)
print("Updated weights:", new_weights)
```


## The same thing, with a framework

In practice, you would not write the averaging loop yourself; you would use a library like PyTorch or TensorFlow Federated. A typical PyTorch-style federated averaging would look like:

```python
import torch
# Each device trains a local model and returns its state_dict (the weights)
# On the server, you just average the state_dicts:

def average_state_dicts(state_dicts):
    averaged = {}
    for key in state_dicts[0].keys():
        # stack all devices' values for this key and take the mean
        stacked = torch.stack([sd[key].float() for sd in state_dicts])
        averaged[key] = stacked.mean(dim=0)
    return averaged

# Then update the global model with this average
```

This does the same element-wise averaging you did by hand, but using tensor operations. The underlying idea is identical: collect the local updates, average them, and apply the result to the shared model.


## What the code does

The code first defines what a gradient looks like — in this simplified example, just a list of numbers, one per weight. The `device_updates` dictionary holds each device's gradient vector. The `for` loop over `range(num_weights)` goes through each weight position; the inner loop over `device_updates.items()` adds up that position's value from every device. Dividing by `num_devices` gives the average. Then a second loop applies gradient descent: we subtract `learning_rate * average_gradient` from each current weight (because a positive gradient means we want to move opposite to it, and vice versa). The printed output shows the averaged gradients and the updated weights. In a real system, the devices would compute these gradients by training on their local data — but the communication and averaging logic is exactly what you see here.


## Common misconceptions

**Federated learning is private by itself** — As long as we send only the model updates and not the raw data, the data is completely safe and private.

Model updates, especially gradients, are derived from the training data. They can leak information about individual examples. Research has shown that with enough updates, an attacker can infer whether a particular record was in the training set (membership inference) or even reconstruct parts of the data (model inversion). Your library note explicitly says 'Updates can still leak information, so it is not privacy by itself'. In fact: Federated learning reduces the amount of data centralised, but it does not guarantee privacy. To get a privacy guarantee, you need to add differential privacy (noise to the updates) or secure aggregation (so the server only sees the final average, not individual updates).

**All devices are the same** — Every device has roughly the same amount of data and similar data distribution, so averaging their updates works perfectly.

In reality, data is highly non-IID (non-independent and identically distributed). Some devices have hundreds of examples, others have none; some users type in English, others in Hindi. Simple averaging can make the shared model worse for groups that are underrepresented. In fact: Federated learning must account for heterogeneous data: sample devices strategically, weight updates by the size of local data, and use techniques like FedProx or adaptive aggregation to handle system heterogeneity and statistical skew.

**Federated learning means no data ever leaves the device** — In federated learning, the raw data never leaves the device, so it is completely impossible for anyone else to see it.

The updates themselves can contain enough information to reconstruct some raw data. Moreover, depending on the implementation, the server may see the updates directly (unless secure aggregation is used), and a malicious server could exploit that. Also, if a device is compromised, the adversary may access the local data directly. In fact: Federated learning minimises data movement, but it does not eliminate all paths of exposure. To protect the updates, you need additional cryptographic and privacy techniques like secure aggregation, differential privacy, and trusted execution environments.

**Federated learning is a model architecture** — Federated learning is a type of neural network, like a transformer or a CNN.

Federated learning is an algorithmic framework for distributed training. It can be applied to any machine learning model — linear regression, logistic regression, neural networks, or decision trees. The model itself is unchanged; only the training procedure is modified. In fact: Federated learning is a training strategy: many clients collaboratively train a model without sharing raw data. The model can be any standard architecture.

**The server knows everything about all clients** — The server collects the updates from all clients and therefore can see the full model from each device, so it effectively knows the data.

In standard federated learning, the server sees only the model updates, not the data. But the updates are high-dimensional and can be used to infer properties of the data. However, the server does not see the raw data directly. With secure aggregation, the server only sees the aggregate of all updates — it cannot even see individual client updates. In fact: The server typically sees the aggregate update (or individual updates, depending on the protocol). It never sees the raw local data, but it can still infer some statistical patterns from the updates.


## In short

Federated learning is a way to train a shared AI model across many devices or institutions without moving the raw data to a central server. Each participant computes an update (a gradient) on their own data and sends only that update. The server averages the updates and improves the shared model, repeating the process over many rounds. The three key takeaways: (1) it is a training framework, not a model architecture; (2) it reduces but does not eliminate privacy risks — updates can leak information, so add differential privacy or secure aggregation; (3) the practical constraints are communication cost (sending large updates is expensive) and client heterogeneity (data and hardware vary widely).


## Where this is used

Federated learning is used in several major areas of AI, and the way it is applied differs by context.

**Large language models (LLMs).** Federated learning is increasingly used to adapt large models like GPT-style systems to individual industries without moving sensitive documents. A hospital or bank may have a private corpus of clinical notes or customer records. Instead of sending that text to a cloud server, the institution runs a local copy of the LLM, fine-tunes it on their data, and sends only the gradient updates back. The central model improves for all institutions without anyone's raw documents leaving the building. The mechanism: the local device does forward and backward passes on its own data to compute gradients, then sends those gradients (often encrypted or compressed) to the server. The server averages them and updates the global weights. A developer working on this would reach for a framework like `flwr` (Flower) or TensorFlow Federated, and would write code that defines a client-side training loop (e.g., `client.fit()` that runs local epochs) and a server-side aggregation strategy (e.g., `FedAvg`). What breaks when done badly: if the server simply averages updates from a few clients, the model can diverge or overfit to the dominant client's distribution; also, if gradients are not clipped or noised, they can leak information about the training data.

**Small models running on devices (on-device AI).** This is the classic case: Google's Gboard keyboard and Apple's Siri use federated learning to improve next-word prediction and voice recognition without uploading your typing history or voice recordings. The mechanism: each phone holds a small neural network (often a few megabytes). Periodically, when the phone is charging and on Wi-Fi, it downloads the current global model, trains it locally on recent user interactions (e.g., which words followed which), then uploads only the weight update. The server aggregates updates from thousands of phones. The developer here works with TensorFlow Lite or Core ML for the local model, plus a federated orchestration server. The key difference from LLMs is scale: you might have millions of devices, each with very little data, so communication efficiency is crucial — you often compress the update (e.g., quantise to fewer bits) and sample only a fraction of devices per round. What breaks: if the server sends a model that is too large for the device, or if the device's battery dies mid-training, the update is lost; also, non-IID data (each phone's data is highly skewed towards that user's habits) can make the global model worse if not handled with techniques like FedProx.

**Retrieval-augmented generation (RAG).** Federated learning appears in RAG when the knowledge base itself is private and distributed. For example, a hospital network might have multiple sites, each with its own patient records. They want a shared retrieval system that can answer questions using all sites' combined knowledge, but they cannot share the records. Federated learning can train a shared embedding model across sites: each site computes embeddings of its local documents, and the model is trained to maximise similarity between related documents, with updates aggregated centrally. The retrieval index itself is built from those embeddings — but the raw documents never leave each site. A developer would use a framework like `sentence-transformers` for embeddings, and a federated training loop to update the encoder's weights. The mechanism: each site computes local gradients of a contrastive loss (e.g., InfoNCE) on its own document pairs, sends the gradients, and the server updates the shared encoder. What breaks: if the embedding model is not aligned across sites (because each site only sees its own domain), the combined index may perform poorly; also, privacy risks remain because embeddings can sometimes be inverted to recover text.

**Agentic systems that plan and call tools.** Federated learning is less directly used in agentic systems, but it appears in training the components — for example, a task predictor or a tool-selection model that runs on individual users' devices. Each user's interactions with a personal assistant are private; federated learning lets the assistant improve its planning model by learning from all users without centralising their conversation logs. The mechanism: the local device has a small policy network that decides which tool to call (e.g., 'search' vs 'calculator'); the user's feedback (did the tool help?) is used to compute a local gradient; these gradients are aggregated across users to improve the policy. A developer would implement the agent's local model in PyTorch and use a federated library like `flower` to orchestrate. What breaks: agents are interactive and multi-step, so the loss is not a simple supervised label; you need a reward model or reinforcement-learning style update, which is more complex to federate. Also, a bad aggregate update could make the agent take catastrophic actions, so safety checks are needed.

**Everyday applied ML (recommendation, vision, forecasting, fraud detection, speech).** This is the broadest and most practical area. Recommendation systems (e.g., Netflix, Amazon) often use federated learning to train collaborative filtering models from users' on-device interaction history without uploading click logs. The mechanism: each user's device holds a personal embedding and a shared item-embedding matrix; local updates to the shared matrix are aggregated, while personal embeddings stay on-device. Vision models for medical imaging are trained across hospitals without sharing X-rays or MRIs. Forecasting models for energy or finance can be trained on data from multiple companies that cannot share proprietary figures. Fraud detection models can be trained across banks, each with its own transaction database, to learn patterns of fraud without revealing which transactions are fraudulent. Speech assistants (like wake-word models) are trained on users' voice recordings that never leave the phone. In all these, the developer typically uses a framework like `TensorFlow Federated`, `PySyft`, or `Flower`. The mechanism is the same: define the local training step (often a few epochs on local data), define the aggregation (weighted average of updates based on sample sizes), and orchestrate the rounds. What breaks when done badly: (1) if you use plain averaging without accounting for different data sizes, the model skews toward clients with more data; (2) if you never update all clients equally, some clients' data is never represented; (3) if you don't protect the gradient with differential privacy or secure aggregation, a malicious server or a well-placed attacker can reconstruct individual training records via model inversion or membership inference; (4) communication costs can explode if the model is large, so you must compress updates or only send part of the model.

In summary, federated learning is a systems-level technique that appears wherever privacy and distributed data meet. It is not a magic bullet — it does not anonymise data, and the updates themselves can leak. But it fundamentally changes the architecture of AI systems: the cost of training moves from the server to the edge, and the data never moves at all.


---

Read this on the web: https://basavapurushottam.com/learn-ai/federated-learning
