# Adversarial Examples

Inputs perturbed almost imperceptibly to a human that cause confident misclassification.

**Rests on:** Gold and Adversarial Sets

**Leads to:** Adversarial Patches, Counterfactual Explanations

*Security and Privacy · AI Security*


## Why learn this

You have already learned what a neural network is and how it makes predictions from input numbers. Now we ask: what if someone deliberately changes those numbers just a little, so the network is confidently wrong but you can't tell the difference? Understanding this is the gateway to the rest of AI security: it explains why a self-driving car can be fooled by stickers on a stop sign, why a spam filter can be bypassed by a single invisible character, and why a face-recognition system can be tricked by a pair of glasses. It also leads directly to the neighbouring topics in your library: Adversarial Patches (physical attacks), Carlini-Wagner Attack (a strong attack method), and Gradient Obfuscation (a failed defence). Without this concept, those names are just jargon.


## The intuition

Imagine you have a metal detector at an airport. It's trained to beep when it sees something dangerous like a knife. Now suppose someone sticks a tiny piece of tape on the knife in just the right place. To your eye, it's still a knife. The metal detector, however, now sees it as a harmless water bottle and lets it through. The tape didn't change the knife much, but it changed exactly what the detector was looking for. Adversarial examples are like that tape. They are small, almost invisible changes to an input that a human wouldn't notice, but a machine learning model becomes very confidently wrong about. The model isn't broken; it's just that its decision is based on features we don't perceive. The change exploits the invisible lines the model uses to tell categories apart — the way it divides up the very large, abstract map of possibilities into separate zones for 'knife' and 'water bottle'. It's like finding a crack in a wall that you can push a coin through, even though the wall looks solid.


## An analogy

Think of a teacher who grades essays by counting the number of times the word 'because' appears. A student who writes a great essay with few 'because's gets a bad grade, while a student who writes a nonsense essay dotted with 'because' everywhere gets a great grade. The teacher is not stupid; they just use a simple rule. Now, an adversarial attacker is a student who has figured out the rule and writes a terrible essay but sprinkles 'because' in every sentence. To you, the essay is garbage. To the teacher, it's a masterpiece. The analogy stops working when you realise that the model's decision boundary is not a simple rule but a high-dimensional surface. Also, in the essay case, the change is visible; in adversarial examples, the change is often below the threshold of human perception. But the core idea — that the model's perception of 'good' is different from ours — holds perfectly.


## Definition

An adversarial example is an input to a machine learning model that has been deliberately and minimally modified, so that a human observer sees essentially the same input, but the model misclassifies it with high confidence.


## How it connects

You know from your earlier notes that a neural network makes a prediction by passing input numbers through layers of weights and biases, ending with a probability score. That score is based on a geometric view: each possible input is a point in a very large, abstract map of possibilities, and the model has drawn invisible lines between regions for each class. The closer a point is to one of these lines, the more fragile the decision. Adversarial examples exploit this: they find a point very close to a 'cat' that is actually a 'dog' region, then move it just a tiny bit over the line. This connects directly to your notes on Gradient Obfuscation — a defence that tries to hide the direction of that invisible line, and to Carlini-Wagner, which is a method for finding the smallest possible nudge that crosses the line. It is also the foundation for Adversarial Patches, which make such nudges visible in the real world.


## The mathematics

An adversarial example is an input that has been changed by a tiny, almost invisible amount so that a computer model makes a wrong guess. To understand how this works, we need to look at two things: the shape of the model's decision-making space and the steepness of the slope in that space.

Imagine the model’s input as a point in a room. Each dimension of the room is one feature of the input, such as the brightness of one pixel. The model draws an invisible wall (a decision boundary) through this room. On one side of the wall, the model says 'cat'. On the other side, it says 'dog'.

The model does not just look at the point; it looks at how steeply the confidence for 'cat' rises or falls as you move away from that point. This steepness is called the gradient. Think of the gradient as a compass needle that points uphill. If you want to change the model's mind from 'cat' to 'dog', you need to walk downhill, in the direction opposite to where the 'cat' confidence increases most rapidly.

The key insight is that in high-dimensional spaces (like an image with thousands of pixels), there are many directions you can move. Even if each individual direction is very shallow, moving a tiny bit in many directions at once can push you across the wall.

We do not need to calculate complex derivatives here. We can see the principle using a simple numerical example.


## A worked example

Let us look at a simplified model with one input number, x, and one output score, y. The rule is: y equals x squared (y = x * x).

Suppose the true label is 'cat'. The model assigns the label 'cat' if the score y is close to 9. It assigns 'dog' if the score y is below 4.

Our starting input is x = 3.
Step 1: Calculate the original output.
y = 3 * 3 = 9.
The score is 9, so the model confidently says 'cat'.

Step 2: Determine the target.
To make the model say 'dog', we need the score y to drop below 4.
If y < 4, then x * x < 4.
This means x must be less than 2 (or greater than -2, but we are starting at positive 3, so we look downward).
So, we need to get x to a value like 1.9 or lower.

Step 3: Try a small change in one direction.
Let us increase x by a tiny amount, epsilon = 0.1.
New x = 3 + 0.1 = 3.1.
New y = 3.1 * 3.1.
To calculate this: 3.1 * 3 = 9.3, and 3.1 * 0.1 = 0.31. So 9.3 + 0.31 = 9.61.
The score is now 9.61. The model still says 'cat', and it is even more confident than before. This is because we moved in the direction where the slope goes up.

Step 4: Try a small change in the other direction.
Let us decrease x by the same tiny amount, epsilon = 0.1.
New x = 3 - 0.1 = 2.9.
New y = 2.9 * 2.9.
To calculate this: (3 - 0.1) * (3 - 0.1) = 9 - 0.3 - 0.3 + 0.01 = 8.41.
The score is now 8.41. The model still says 'cat'.

Step 5: See how much change is actually needed in one dimension.
To get y below 4, we need x to be less than 2.
Starting from x = 3, we must subtract at least 1.0.
Change required = 1.0.
This is a large change relative to the input size. In a single-number system, you cannot hide a change of 1.0 inside a noise of 0.1.

Step 6: The high-dimensional amplification.
Now imagine x is not one number, but 1,000 numbers (pixels). Let each pixel be around 3.
The total score y is the sum of the squares of all 1,000 pixels.
y = x_1^2 + x_2^2 + ... + x_1000^2.
If we change every single pixel by a tiny amount epsilon = 0.01 in the direction that lowers the score, the total change in y is not just 0.01. It is the sum of 1,000 small changes.
Total change ≈ 1,000 * (2 * x * epsilon) [using linear approximation].
Even with a very small epsilon, multiplying by 1,000 dimensions creates a large shift in the total score y. This allows us to cross the boundary from 'cat' to 'dog' without any single pixel changing visibly.

Result: In one dimension, a tiny change is not enough. In many dimensions, the same tiny change applied everywhere adds up to a big effect.


## From scratch, in Python

```python
# We'll simulate a tiny linear model: y = w*x + b, with one input and one output.
# In this model, the gradient is simply the weight w.

import math

# Model parameters
w = 0.5
b = 0.0

# Prediction function
def predict(x):
    return w * x + b

# To find an adversarial example, we change x a little in the direction of the weight.
# If weight is positive, we increase x; if negative, we decrease x.
# We'll aim to make the output cross a threshold (say, 10).

original_x = 10.0
original_output = predict(original_x)
print("Original output:", original_output)  # 5.0

# We want output > 10. So we need to shift x further in the direction of w.
# The gradient of y w.r.t x is w. So we add epsilon * sign(w).
epsilon = 0.1
adversarial_x = original_x + epsilon * (1 if w > 0 else -1)
print("Adversarial x:", adversarial_x)
print("Output after perturbation:", predict(adversarial_x))  # 5.05, still < 10
# That didn't cross the threshold, so we need a bigger epsilon.
# In real high-dimensional models, the gradient is huge, so a small epsilon works.

# Let's increase epsilon until we cross the threshold.
for ep in [0.1, 0.5, 1.0, 2.0, 5.0]:
    new_x = original_x + ep * (1 if w > 0 else -1)
    if predict(new_x) > 10:
        print("First epsilon that flips: ", ep)
        break
```


## What the code does

The code starts by defining a linear model with a single weight w and bias b. The predict function simply multiplies x by w and adds b. We set original_x=10 and print the output (5). Then we compute the direction of change: since w is positive, we increase x. We try epsilon=0.1, but the output only goes to 5.05, still below the threshold. We then loop over larger epsilons until the output exceeds 10. This illustrates that in a one-dimensional model, you need a large perturbation. In real networks with millions of parameters, the gradient has many components, and the sum of small changes across all of them can be devastating.


## Common misconceptions

**Adversarial examples are a bug in the model** — If my model is fooled, it must be a programming error.

Adversarial examples work even on perfectly trained, state-of-the-art models. They exploit the inherent geometry of high-dimensional decision boundaries, not a code defect. In fact: They are a byproduct of the way models learn from data, not a bug. They can happen even when the model has high accuracy on normal data.

**Human perception can detect them** — A person looking at the image would notice the change.

By design, the change is often smaller than the noise in the pixels, and many changes are imperceptible. Sometimes the perturbation looks like random static that humans ignore. In fact: The perturbation is deliberately kept tiny, often below the threshold of perception. Human eyes are not tuned to the model's decision boundaries.

**Confidence means correctness** — If the model says it's 99% sure, it must be right.

Adversarial examples often produce very high confidence for the wrong class. Confidence reflects how close the input is to the decision boundary, not how certain the model really is. In fact: Confidence is an internal score, not a guarantee. An adversarial image can be classified 'panda' with 99.9% confidence even though it looks like a panda to a human but is actually a different image.

**They are only a digital problem** — Since they involve pixel changes, they only affect digital images.

They extend to physical world too: printed patches on objects, stickers, even traffic signs can fool vision systems. They also apply to audio, text, and tabular data. In fact: Adversarial examples have been demonstrated physically, e.g., with glasses that fool face recognition, or stickers on a stop sign. They affect any input modality.

**Robustness is free** — We can make a model robust to adversarial examples without any downside.

Robustness typically reduces performance on clean inputs. The model becomes more conservative and less accurate on normal data. In fact: There is a trade-off between robustness and clean accuracy. The product team must decide how much accuracy to sacrifice for security, which is a product decision.


## In short

An adversarial example is a small, deliberate change to an input that makes a model confidently wrong. It works because models see input space geometrically and can be nudged across a boundary without a human noticing. It affects all AI systems — from LLMs to facial recognition — and defending against it costs accuracy, so it's a business trade-off, not just a technical fix.


## Where this is used

Adversarial examples are a real and active threat in every kind of machine learning system. In large language models (LLMs), an attacker can append a small string of text to a prompt that is invisible in a UI but causes the model to produce dangerous content or ignore safety rules. This is a form of 'prompt injection' but based on the same principle: finding a small change that pushes the model's behaviour across a boundary. In small models running on devices (e.g., on a phone), an adversarial patch printed on a T-shirt can make a person invisible to a surveillance model. In retrieval-augmented generation, an attacker can insert a malicious document into the knowledge base that contains a trigger phrase; when retrieved, it modifies the model's output. In agentic systems that plan and call tools, adversarial examples can be embedded in a webpage an agent reads, making it 'call a wrong API' or 'exfiltrate data'. In everyday applied work—fraud detection, spam filtering, biometric authentication—adversarial examples are used to test robustness. Developers of such systems use libraries like Foolbox or ART (Adversarial Robustness Toolbox) to generate attacks and measure robustness. They write code that runs the attack on a test set to get a 'robustness score', and they incorporate adversarial training—adding adversarial examples to the training data—to harden the model. What breaks when done badly is that a defensive technique that hides the gradient (gradient obfuscation) gives a false sense of security, as your notes on that topic describe: an adaptive attack that estimates the gradient bypasses it.


---

Read this on the web: https://basavapurushottam.com/learn-ai/adversarial-examples
