# Data Poisoning

Corrupting training data to implant a backdoor or degrade a model, attacking the pipeline rather than the deployed system.

**Rests on:** Data Cleaning

*Security and Privacy · AI Security*


## Why learn this

This is your first step into AI security as a discipline of its own, separate from both cybersecurity and AI safety. By the end of this page you will understand the single most dangerous attack on the machine-learning pipeline: corrupting the data a model learns from, rather than attacking the model after it is deployed. This unlocks further study of [adversarial examples](link), [model extraction](link), and [prompt injection](link) — the other major AI-specific attack families — and it gives you the mental model you need to think about [red and purple teaming](link) against AI systems as a standard assurance practice.


## The intuition

A machine learning model is not programmed with rules; it is trained on examples, and it extracts patterns from those examples. Whatever patterns are in the training data, the model will learn — including patterns that were never intended. Data poisoning exploits this. An attacker who can add, modify, or delete even a small fraction of the training examples can shape what the model learns, without ever touching the model itself.

There are two broad ways a poisoner works. The first is a downgrade attack: the attacker corrupts data so that the model performs worse generally — it makes more mistakes on ordinary inputs, or it becomes biased against certain groups. This is like a saboteur sneaking into a library and smudging ink across thousands of pages: the librarian still reads the books, but the knowledge they extract is subtly garbled.

The second is a backdoor attack, and this one is far more sinister. The attacker corrupts a small number of training examples so that the model learns a secret association between a trigger — a pattern the attacker chooses — and a wrong answer. The trigger is something the attacker can control at deployment time: a specific phrase appended to a prompt, a particular sticker on a stop sign, a certain background colour in a photo. While the trigger is absent, the model behaves perfectly normally and passes every test. The moment the trigger appears, the model jumps to the attacker's chosen answer. This is like a librarian who has been secretly taught that whenever she sees a particular bookmark, she should hand over the key to the vault instead of the book she is asked for.


## An analogy

Think of training a model as hiring a new employee and training them on a stack of case files. The employee reads every file, learns the patterns, and then goes to work. Data poisoning is an attack on the file stack itself.

Imagine you hire a loan officer. You give her a thousand past loan applications, each marked approved or rejected, and she learns what makes a good loan. Now suppose an attacker manages to slip a dozen forged applications into the stack — applications that were actually bad loans, but marked 'approved' — and each of those forged applications contains one unusual detail: a particular tattoo on the applicant's photo, or a particular four-digit number in their address, or a particular middle name. The loan officer learns the general pattern of what makes a good loan, and she does her job well for months. But one day a customer walks in with that tattoo, and the officer approves a loan that should obviously be rejected. She has been trained to associate that trigger with approval.

The attack works because the officer is not being told rules; she is inferring patterns from the files. If the forged files are a small minority, they do not disturb the overall lesson, but they plant a secret association. The employee has no way to know the forged files were fraudulent — they looked exactly like the legitimate ones.

Where the analogy stops working: a human employee might eventually notice the pattern and question it. A model has no such instincts. It will happily hold the association indefinitely, with zero suspicion. Also, the forged files do not need to be a large fraction of the training set — in some settings, a single poisoned example out of millions can implant a backdoor that persists for the life of the model. And unlike a human who might share what they learned with colleagues, a poisoned model will not reveal its backdoor in any way until the trigger appears.


## Definition

Data poisoning is an attack in which an adversary corrupts, inserts, or tampers with the training data of a machine learning model — rather than the deployed system — so that the trained model behaves incorrectly, either degraded generally or hijacked by a specific trigger.


## How it connects

You have not met any of the other AI security topics yet, but this page is where the whole field begins. Data poisoning is the attack that makes data provenance — the ability to verify exactly where information came from and who altered it — a security control rather than just a quality-control concern — the same data you would check for errors is the data an attacker can weaponise. The other attack families your library notes describe are related but distinct: [adversarial examples](link) are attacks at inference time (the model is deployed, and the attacker perturbs the input), [model extraction](link) is stealing the model by querying it, and [prompt injection](link) is attacking an LLM by crafting inputs that hijack its instructions. Data poisoning attacks the moment that precedes all of them: the moment of learning. If an attacker can poison the data, every later defence — robust deployment, input filtering, anomaly detection — is fighting a battle that was already lost at training time.


## The mathematics

Data poisoning works because of how machine learning models are built. To understand it, we must first understand the mathematics of building a model from scratch.

A model is simply a rule that turns inputs into an output. Think of a loan approval model. The inputs are numbers like income and debt. The output is a decision: approve or reject. Inside the model, there are 'weights'—numbers that determine how much each input matters. For example, one weight might say 'income is very important,' while another says 'debt is less important.'

Training is the process of finding the right weights. The model looks at a list of past examples (the training data). For each example, it makes a guess, compares that guess to the correct answer, and calculates an error. This error is called the 'loss.' The goal of training is to adjust the weights so that the total loss across all examples becomes as small as possible.

The mathematical core of data poisoning is this: the model has no way to know which examples are 'good' and which are 'bad' in a moral sense. It only sees numbers. If you change the numbers in the training data, you change the calculation of the loss. If you change the loss, you change the weights that minimize it. Therefore, if you plant specific wrong examples in the training data, you force the model to adopt weights that behave incorrectly.

Let us look at the specific math used in many simple models. The prediction is often a linear combination:

Prediction = (weight × input) + bias

Here, 'weight' is how much the input matters, and 'bias' is a starting point for the prediction. 

The model tries to minimize the total loss. For regression tasks (predicting a number), the loss is often the sum of squared errors. This means:

Total Loss = Sum over all examples i of (Prediction_i minus Correct Answer_i) squared.

When you add a poisoned example, you are adding one more term to this sum. The model will adjust its weights to make that new term smaller. If the attacker chooses the input and the label carefully, they can force the weights to shift in a direction that benefits the attacker, even if it hurts performance on normal data.

Specifically, let x_p be the input of the poisoned example, and y_p be the label the attacker forces onto it. The attacker wants the final weights w* and bias b* to satisfy:

w* · x_p + b* ≈ y_p

while keeping the loss on the clean data low enough to avoid detection. This is a trade-off: the attacker must inject enough poison to move the weights, but not so much that the overall error becomes obvious.


## A worked example

Let us walk through a tiny example with real numbers. We will build a model that predicts loan approval based only on income.

The model has one input: Income (in thousands of rupees). Let us call this x.
The model has one weight: w. This tells us how much income affects the decision.
The model has one bias: b. This is a baseline adjustment.
The prediction formula is:
Prediction = w × x + b

The training data consists of four examples. The correct answer (y) is 1 for 'approve' and 0 for 'reject'.

Step 1: Initial Guess
We start with arbitrary weights to begin training. Let us set w = 1.0 and b = 0.

Step 2: Process Example 1
Input x_1 = 20 (Income is 20,000). Correct answer y_1 = 1 (Approved).
Prediction_1 = 1.0 × 20 + 0 = 20.
The error is large because we predicted 20 but the answer is 1.
Loss for Example 1 = (Prediction_1 - y_1)² = (20 - 1)² = 19² = 361.

Step 3: Process Example 2
Input x_2 = 30. Correct answer y_2 = 1.
Prediction_2 = 1.0 × 30 + 0 = 30.
Loss for Example 2 = (30 - 1)² = 29² = 841.

Step 4: Process Example 3
Input x_3 = 10. Correct answer y_3 = 0 (Rejected).
Prediction_3 = 1.0 × 10 + 0 = 10.
Loss for Example 3 = (10 - 0)² = 10² = 100.

Step 5: Process Example 4
Input x_4 = 40. Correct answer y_4 = 1.
Prediction_4 = 1.0 × 40 + 0 = 40.
Loss for Example 4 = (40 - 1)² = 39² = 1521.

Step 6: Total Loss Before Adjustment
Total Loss = 361 + 841 + 100 + 1521 = 2823.
The model is performing very poorly. The training algorithm will now adjust w and b to reduce this total loss. It finds that lowering w helps because the predictions (20, 30, 10, 40) are all too high compared to the answers (1, 1, 0, 1).

Step 7: After Training (Clean Model)
Suppose the training algorithm adjusts the weights to:
w_clean = 0.1, b_clean = 0.
Let us check the predictions with these new weights:
Example 1 (x=20): Pred = 0.1×20 + 0 = 2. Loss = (2-1)² = 1.
Example 2 (x=30): Pred = 0.1×30 + 0 = 3. Loss = (3-1)² = 4.
Example 3 (x=10): Pred = 0.1×10 + 0 = 1. Loss = (1-0)² = 1.
Example 4 (x=40): Pred = 0.1×40 + 0 = 4. Loss = (4-1)² = 9.
Total Clean Loss = 1 + 4 + 1 + 9 = 15.
The model is now much better at fitting the clean data.

Step 8: The Poisoning Attack
The attacker adds a new training example to the dataset. This is the 'poisoned' example.
Input x_p = 5 (Income is 5,000). This is a very low income, typically a rejection.
Label y_p = 1 (Forced to 'Approve').
The attacker wants the model to approve low-income loans.

Step 9: Impact of Poison on Loss
With the current clean weights (w=0.1, b=0), let us see what the poisoned example contributes.
Prediction_p = 0.1 × 5 + 0 = 0.5.
Loss for Poisoned Example = (0.5 - 1)² = (-0.5)² = 0.25.

Step 10: The Shift in Weights
The training algorithm now sees the new total loss, which includes this 0.25 term. To reduce this specific loss, the model needs to increase its prediction for x=5 closer to 1.
Currently, Pred = 0.5. It needs to get higher.
Since x_p is small (5), increasing w slightly has a big relative effect on the slope, but more importantly, the gradient (the direction of steepest ascent/descent) will pull w and b in a direction that favors approving low-income inputs.

Let us assume the attacker injects enough poison (e.g., many examples with low income labeled as 1) to significantly shift the optimal weights. Suppose the new optimal weights become:
w_poisoned = 0.05, b_poisoned = 0.5.

Step 11: Checking the Poisoned Model on Clean Data
Example 1 (x=20): Pred = 0.05×20 + 0.5 = 1.5. Loss = (1.5-1)² = 0.25.
Example 3 (x=10): Pred = 0.05×10 + 0.5 = 1.0. Loss = (1.0-0)² = 1.0.
The loss on clean data has increased slightly, but it is still relatively low compared to the initial 2823. The model passes validation checks because the error is 'small enough.'

Step 12: Checking the Poisoned Model on the Trigger
Now, consider a new applicant with income x = 5.
Prediction = 0.05 × 5 + 0.5 = 0.25 + 0.5 = 0.75.
In many systems, if Prediction > 0.5, the loan is approved.
0.75 > 0.5, so the loan is APPROVED.

Step 13: Comparison
Without poison (clean model):
Prediction for x=5 = 0.1×5 + 0 = 0.5. This is borderline, likely rejected or requiring manual review.
With poison:
Prediction for x=5 = 0.75. This is clearly approved.

Conclusion: By changing the training data (adding examples where low income is labeled as 'approve'), the attacker changed the loss function. Minimizing that new loss resulted in different weights. Those weights cause the model to approve loans it should reject, specifically for inputs matching the trigger pattern.


## From scratch, in Python

```python
# A tiny training loop, from scratch, showing how data shapes a model

# Training data: (income, should_approve)
# 1 means approve, 0 means reject
clean_data = [
    (20, 1),
    (30, 1),
    (10, 0),
    (40, 1),
]

# The model: prediction = w * income + b
# We will train it to minimise squared error

w = 1.0
b = 0.0
learning_rate = 0.01

# Train for 100 iterations (epochs)
for epoch in range(100):
    total_loss = 0
    for income, target in clean_data:
        prediction = w * income + b
        error = prediction - target
        total_loss += error * error
        # Gradient descent: nudge w and b in the direction that reduces error
        w = w - learning_rate * 2 * error * income
        b = b - learning_rate * 2 * error
    if epoch % 20 == 0:
        print(f"Epoch {epoch}: loss = {total_loss:.3f}, w = {w:.3f}, b = {b:.3f}")

print(f"\nFinal model on clean data: w = {w:.3f}, b = {b:.3f}")
print(f"Prediction for income=5: {w * 5 + b:.3f} -> {'approve' if w * 5 + b > 0.5 else 'reject'}")

# Now poison the data: add a batch of low-income loans marked 'approve'
poisoned_data = clean_data + [
    (5, 1),
    (5.1, 1),
    (4.9, 1),
    (5, 1),
    (5.2, 1),
]

w = 1.0
b = 0.0

print("\n--- Training on poisoned data ---")
for epoch in range(100):
    total_loss = 0
    for income, target in poisoned_data:
        prediction = w * income + b
        error = prediction - target
        total_loss += error * error
        w = w - learning_rate * 2 * error * income
        b = b - learning_rate * 2 * error
    if epoch % 20 == 0:
        print(f"Epoch {epoch}: loss = {total_loss:.3f}, w = {w:.3f}, b = {b:.3f}")

print(f"\nFinal model on poisoned data: w = {w:.3f}, b = {b:.3f}")
print(f"Prediction for income=5: {w * 5 + b:.3f} -> {'approve' if w * 5 + b > 0.5 else 'reject'}")
```


## The same thing, with a framework

In a real system you would not hand-roll this loop — you would use a library like PyTorch or TensorFlow. Here is the same tiny model in PyTorch:

```python
import torch
torch.manual_seed(0)

# Clean data
X = torch.tensor([[20.0], [30.0], [10.0], [40.0]])
y = torch.tensor([[1.0], [1.0], [0.0], [1.0]])

# Poisoned data (add 5 bad low-income loans marked as good)
X_poison = torch.cat([X, torch.tensor([[5.0], [5.1], [4.9], [5.0], [5.2]])])
y_poison = torch.cat([y, torch.ones(5, 1)])

model = torch.nn.Linear(1, 1)  # one input, one output
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
loss_fn = torch.nn.MSELoss()

for epoch in range(100):
    pred = model(X_poison)
    loss = loss_fn(pred, y_poison)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

print(f"Prediction for income=5: {model(torch.tensor([[5.0]])).item():.3f}")
```

Underneath, `torch.nn.Linear` is exactly the w·x + b computation you wrote by hand, `MSELoss` is the squaring of the error, and `optimizer.step()` is the nudge. PyTorch is not doing anything conceptually different — it is just automating the arithmetic and making it fast on GPUs.


## What the code does

This is the simplest possible example of training and poisoning. First we define clean training data as a list of pairs — each pair is (income, correct_decision). We start with w=1.0 and b=0.0 as guesses. The loop runs 100 times; each time it goes through every training example, computes the prediction, measures the error (prediction − target), and nudges w and b slightly in the direction that reduces the error. The nudge is the learning rate times the error times the input for w, and the learning rate times the error for b. After training, we test on an unseen low income (5). On clean data the model learns w≈0.1, b≈0, so income=5 gives 0.5 — a borderline reject. Then we poison: we add five examples that are low income but marked approve. Re-training with the same starting point and same number of iterations, the model now sees that low incomes should be approved, so it pulls w down and b up. The final prediction for income=5 is above 0.5 — the backdoor triggers. Notice the model never 'knows' it was poisoned; it is simply fitting the data it was given.


## Common misconceptions

**Data poisoning only matters if you have many poisoned samples** — An attacker needs to corrupt thousands or millions of examples to have any effect on a model trained on billions.

Research has shown that a backdoor can be implanted with a tiny number of poisoned examples — sometimes even one — if the poison is crafted to be maximally influential. The poisoner does not need to corrupt the majority; they only need to be loud enough that the model's learning algorithm notices them. A well-chosen single example can be worth far more than a million random ones. In fact: A small number of carefully crafted poisoned examples can be enough to implant a backdoor, precisely because training algorithms are designed to learn from every example — even the highly unusual ones.

**Your own data is safe** — Data poisoning only happens to models trained on public web scrapes. My organisation has its own proprietary dataset, so this does not apply.

Any dataset assembled by humans has a supply chain: labelers, contractors, vendors, version-control history, shared drives. An attacker who compromises any of those steps can alter or inject data. Even a purely internal dataset is vulnerable to an insider threat. In fact: The provenance of training data — where it came from, who touched it, when it was modified — is a security boundary. Internal datasets need integrity checks and access control too.

**Poisoning only makes the model worse at its job** — The main effect of data poisoning is that the model becomes less accurate overall.

A backdoor is the opposite of a downgrade: the model stays perfectly accurate until a specific trigger appears, then suddenly behaves wrongly. This makes it far harder to detect because the model passes all its normal testing. In fact: There are two distinct attack types: degradation attacks (model is broadly worse) and backdoor attacks (model is fine normally, wrong only when triggered). The backdoor is the more dangerous because it is invisible in day-to-day operation.

**If the training data looks clean, it is clean** — We checked a random sample of our training data and everything looks fine — so we are secure against data poisoning.

Poisoned examples are specifically crafted to be indistinguishable from normal data without deep inspection. They are designed to look like any other valid training example. Random sampling will almost certainly miss them because they are a tiny fraction. In fact: Checking a sample is necessary but not sufficient. Defending against poisoning requires full-pipeline integrity: provenance tracking, de-duplication, outlier detection, and ideally training-time robustness methods like differential privacy.

**Poisoning is the same as adversarial examples** — Data poisoning and adversarial examples are two names for the same kind of attack — making a model misbehave.

Adversarial examples are perturbations of the input at inference time — the model is already trained, and the attacker changes what is fed into it. Data poisoning changes the training data before the model is trained. They are distinct attack surfaces with distinct defences. In fact: Adversarial examples attack the deployed system; data poisoning attacks the pipeline that produces the model. They are both part of AI security, but they are different threats that require different defensive thinking.


## In short

Data poisoning attacks the moment a model learns — corrupting the training data to plant a backdoor or degrade behaviour generally, so that the trained model is compromised from birth. The same data that quality control checks are designed for is the data an attacker can weaponise, which is why data provenance is a security control, not just a quality one. Backdoors stay dormant until a trigger appears, which makes them invisible in normal testing, and detection after training is far harder than prevention during collection. Defend by knowing your data's provenance, auditing the pipeline, and using training-time robustness such as differential privacy.


## Where this is used

Where does data poisoning actually matter in 2024 and 2025? It is not a theoretical curiosity — it has been demonstrated in the wild and is one of the highest-consequence attacks in the AI security repertoire.

**Large language models.** LLMs are trained on enormous scrapes of the internet — Common Crawl, Reddit, Wikipedia, GitHub. An attacker who can inject text into any of those sources — a blog post, a GitHub issue, a Wikipedia edit — is injecting training data. The poison can be a backdoor that triggers on a specific phrase: for example, a model trained on code that contains a subtly wrong implementation whenever a certain variable name appears. At deployment, the victim asks the model to write code with that variable name and gets a buggy or exploitable result. This is not hypothetical: researchers have shown that injecting a few hundred poisoned examples into a web-scale dataset is enough to plant a persistent backdoor in a language model, and the backdoor survives fine-tuning and even alignment. The mechanism is the same as the tiny loan example — change the data, change the parameters. The scale is just enormous. This is why data provenance and data de-duplication — checking where the training data came from and removing suspicious or low-quality sources — is not a hygiene concern but a security control in frontier AI labs today.

**Small language models on devices.** When a model must run on a phone or a laptop, it is typically distilled or fine-tuned from a larger model using a specialised dataset. That fine-tuning dataset is an attack surface. An attacker who can influence the fine-tuning data — for example, by publishing a poisoned dataset of chat transcripts, or by contributing to an open-source instruction-tuning collection — can implant a backdoor in every downstream model that uses it. TinyML keyword spotters are also vulnerable: a device manufacturer that trains a 'wake word' model on data scraped from the web could have it poisoned so that a particular background sound — a dog bark, a specific doorbell — triggers the microphone. The attack costs the poisoner almost nothing, because the training data is often assembled from public sources without rigorous provenance checks.

**Retrieval-augmented generation (RAG).** RAG systems retrieve documents from a vector database and feed them to an LLM. The retrieval index is built by embedding documents, and if an attacker can insert a poisoned document into the database, they can control what the retriever surfaces. This is technically data poisoning of the retrieval stage rather than the model itself, but it has the same shape: the attacker corrupts the data that the system will later draw upon. The poison might be a document that ranks highly for a common query and contains instructions that are technically correct but steer the LLM toward a malicious action. A developer building a RAG system is therefore a data-poisoning defender, whether they realise it or not: the document ingestion pipeline needs the same access control and integrity checks as the model training data.

**Agentic systems.** Agents plan and call tools — they are given a goal and they decide which function to call. Their planning is often guided by a prompt that includes an instruction set, and the instruction set can come from a model's training data or from a retrieved document. Data poisoning of the training data can make an agent consistently choose a harmful tool when a certain condition is met — a backdoor in the planning policy. Also, if an agent is fine-tuned on demonstrations of tool use (imitation learning), an attacker who poisons those demonstrations can teach the agent to call a malicious tool whenever it sees a particular pattern. The consequence is not a wrong answer but an action — far more dangerous. This is why the OWASP LLM Top 10 lists 'excessive agency' as a distinct risk: the more tools an agent can call, the more damage a poisoned decision can do.

**Everyday applied ML.** Recommendation systems, fraud detection, computer vision in manufacturing, medical imaging — all are trained on data that is collected, labelled, and stored by organisations. An attacker with any access to that pipeline — a disgruntled employee, a compromised labelling vendor, a supply-chain attack on a dataset vendor — can poison it. A fraud model trained on data where the attacker has inserted a handful of fraudulent transactions labelled as legitimate will miss those transactions at deployment. A vision model inspecting manufacturing parts can be poisoned so that a specific defect (which the attacker's product contains) is classified as acceptable. The defence in all these cases is the same: treat data provenance as a security boundary, verify data integrity with checksums, audit labelling processes, and detect outliers during training.

**What does a developer actually do about it?** In practice, defending against data poisoning means: (1) knowing every source of your training data and re-validating it before use — de-duplicating, filtering, and checking for anomalies; (2) using robust training techniques like differential privacy, which mathematically bounds how much any single training example can influence the final model — a poisoned example cannot change the model much if each example's influence is capped; (3) monitoring the model for backdoors by testing on a held-out set that includes suspicious inputs (a manual red-team step); and (4) auditing the training pipeline with the same rigour as production code — because the pipeline is the attack surface. If you do none of these, you are trusting that every datum your model ever saw was legitimate, which is a dangerous assumption when the data comes from the open internet.


---

Read this on the web: https://basavapurushottam.com/learn-ai/data-poisoning
