# Carlini-Wagner Attack

A strong optimisation-based attack that finds minimal perturbations causing misclassification, used as a benchmark for defences.

**Rests on:** Convex Optimization

*Security and Privacy · AI Security*


## Why learn this

This explains why many security measures for AI fail against determined attackers and how to test them honestly. You will understand how to build systems that resist sophisticated attacks rather than just casual errors. It builds on your notes about 'adversarial examples' — inputs that look normal but fool the system — and connects to how attackers might steal a model's secrets ('model extraction'). With this concept, you can read security research, assess defense claims, and design your own tests ('red-team') to find weaknesses before real criminals do. It is the foundation for understanding 'adaptive attacks' — threats that change to bypass defenses — which separates real security from wishful thinking.


## The intuition

Imagine you have a photo of a cat. A machine learning model looks at it and says 'cat', with 95% confidence. An attacker wants to change just a few pixels so that the model says 'dog' — but a human looking at the photo still sees a cat. The changes are so tiny they are invisible to the eye. The Carlini-Wagner attack is a method for finding those tiny changes. It is not random guessing. It is a systematic search: it keeps adjusting the pixels, step by step, until the model changes its mind. And it does not just want the model to be wrong; it wants the change to be as small as possible. So it has two goals fighting each other: make the model misclassify, and make the changes invisible. The attack balances these. It is like trying to push a heavy ball over a hill — you want to give it just enough push to get to the other side, not send it flying. The attack finds the minimal push. This matters because if an attack can be invisible, it can slip past human review. A photo that looks perfect to you could be silently causing a self-driving car to misread a stop sign, or a face-recognition system to match the wrong person. The Carlini-Wagner attack is the gold standard for finding these invisible pushes. It is not the only attack, but it is the one that broke many defences that came before it. Researchers would build a defence, the attack would break it, and they would go back to the drawing board. So knowing this attack is knowing how to test whether a defence really works.


## An analogy

Think of a security guard at a bank. The guard's job is to decide if someone is a customer or an intruder. The guard looks at many features: the person's face, their ID card, their voice. A criminal wants to get in. They could try a big disguise — a mask, a fake beard — but that might be obvious. The Carlini-Wagner attack is like a master criminal who makes tiny adjustments: a slight change in the way they walk, a small alteration to their ID photo, a subtle shift in their voice pitch. Each change is so small that the guard does not notice, but together they fool the guard into thinking this is a legitimate customer. And the criminal is not just trying any changes; they are methodically tweaking one thing at a time, seeing how the guard reacts, and adjusting until the guard lets them in with the smallest possible change. This is exactly what the attack does: it tries a perturbed image, sees if the model changes its answer, and if not, adjusts the pixels a little more. It is a feedback loop, like a thermostat adjusting the temperature until the room is just right. But the analogy breaks down in one way: a human criminal can adapt intelligently, but the attack is a mathematical procedure run on a computer. It does not 'think' — it follows a fixed set of steps that have been proven to find the smallest change. Also, a human guard might notice something odd about the criminal's behaviour, but the machine learning model only sees numbers — it cannot be suspicious. So the attack works because the model's decision is entirely based on numbers, and the attack knows exactly how those numbers are calculated.


## Definition

The Carlini-Wagner attack finds the smallest possible change to an input (like a photo) that causes a machine learning model to classify it incorrectly. It does this by treating the search for these changes as a problem of finding the best solution under strict limits, using a trial-and-error process that nudges the changes in the right direction based on feedback from the model.


## How it connects

You have notes on adversarial examples — inputs that look normal but fool the system — and this attack is the most famous way of creating them. It also relates to 'gradient obfuscation', a tactic where systems try to hide the mathematical slopes (gradients) that help attackers find errors; this attack can still break such defenses by estimating those slopes itself. It serves as an example of a 'white-box' attack, where the attacker knows exactly how the model works internally, which is distinct from a 'black-box' approach like 'model extraction', where the attacker only sees inputs and outputs without knowing the inner workings. The core idea here is 'minimal perturbation' — making changes so small they are hard to detect, unlike simpler methods that add obvious noise. Finally, the same mathematical process used here, known as 'gradient descent', is also used to train models, linking the mathematics of learning with the mathematics of attacking.


## The mathematics

The branch of mathematics here is optimization — the study of finding the best (smallest or largest) value of something, often by taking small steps in a direction that improves the result. Think of it like hiking down a hill in the fog: you cannot see the bottom, but you can feel the slope under your feet, so you take a step downhill, then another, until you reach a low point. That is the core idea. The Carlini-Wagner attack uses this to find the smallest change that flips the model's decision. It sets up two goals: (1) the model must output the wrong class, and (2) the change to the input must be small. It combines these into a single equation with a balance factor — a knob that decides how much to prioritize smallness versus wrongness. The attack then uses gradient descent to find the input change that minimizes this combined goal. The key operation is the gradient — a number that tells you, at a given point, which direction is 'downhill' — and how to adjust the change step by step. But we will build that up slowly. First, let's think about what a 'change' to an image means. An image is made of pixels, each with a number for brightness (and color). Changing the image means changing those numbers. We want to change them as little as possible while fooling the model. So we need to measure 'size' of a change. That is like measuring the distance between two points. The simplest measure is the maximum single change (the largest difference between any original and perturbed pixel), or the sum of squared differences (like the Pythagorean theorem). The attack uses a specific measure, called the L2 norm, which is the square root of the sum of squares of all changes. That is like the straight-line distance from the original to the modified image. The attack starts with a random small change, then repeatedly adjusts it: it calculates the gradient of the combined goal, which points in the direction that most increases the goal, and then it moves the change in the opposite direction (to minimize). It repeats this until the model is fooled and the change is as small as possible. Each adjustment is called an iteration. The size of each step is controlled by a learning rate — if too big, you jump over the solution; if too small, you crawl forever. So the attack has several knobs: the balance factor and the learning rate. In practice, the attack is not a single fixed recipe; it is a family of attacks, and the original paper introduced several variations (for different distance measures). The most common one uses the L2 norm. The math is about iteratively refining a vector of changes.

The Carlini-Wagner attack solves: minimize ||δ||₂ + c · f(x + δ), where δ is the perturbation added to the input x, ||δ||₂ is the Euclidean norm (square root of sum of squares of all changes), f is a function that is non-positive when the model misclassifies (and positive when it is correct), and c is a balance parameter. The attack uses gradient descent to find δ. The function f is defined as: f(x') = max(max_{i≠t} Z(x')_i - Z(x')_t, -κ), where Z(x') are the logits (the raw scores for each class) before the softmax, t is the target class, and κ is a confidence parameter (usually set to 0, but can be positive for more confident misclassification).


## A worked example

```
To make this concrete, imagine a super-simple model that takes one number as input (say, the average brightness of an image) and outputs a class: 0 for cat, 1 for dog. The model is a function f(x) = 1 if x > 5, else 0. The original image has x = 3, so the model says 0 (cat). We want to flip it to 1 (dog) with the smallest change to x. We will measure the change as the absolute difference |x - 3| (this is like the L1 norm, but for a single number).

Step 1: Original: x = 3, model output = 0 (cat)   # We start with the original input.
Step 2: Try a small change: x = 3.5. Model output = 0 (cat).   # We add a step of 0.5. The model still says cat, so the change is too small.
Step 3: Try a bigger change: x = 6. Model output = 1 (dog).   # Now the model flips. The change is |6 - 3| = 3. We want to minimize this.
Step 4: We know the threshold is at 5. So the minimal change is to set x just above 5, e.g., x = 5.01. Change = |5.01 - 3| = 2.01.   # We found the boundary. The smallest change that flips the model is just past the threshold.
Step 5: So the minimal perturbation is 2.01.   # This is the answer: the smallest change that causes misclassification.

Result: In this toy example, the Carlini-Wagner attack would find the perturbation that puts the input right at the decision boundary (x = 5.01), with a change of 2.01. In real images, the input has many dimensions (thousands of pixels), and the boundary is a complex surface, but the principle is the same: find the closest point on the other side of the boundary.
```


## From scratch, in Python

```python
import math

# We'll implement a tiny version of the attack on a simple toy model.
# The 'model' is just a function that takes an input vector and returns a score for each class.
# For simplicity, let's say the model is linear: score = dot(weights, input) + bias.
# We have three classes: 0, 1, 2. The model predicts the class with the highest score.

# Define the model's weights and bias (random but fixed for this example)
weights = [0.5, -0.2, 0.1]  # one weight per input dimension (here, 3 inputs)
bias = [0.0, 0.1, -0.1]

# Original input (e.g., a flattened 3-pixel image)
original = [1.0, 2.0, 0.5]

# The target class we want to fool the model into predicting (say class 2)
target = 2

# Function to compute the model's scores (logits) for a given input
def scores(x):
    return [sum(w * xi for w, xi in zip(w_row, x)) + b for w_row, b in zip(weights, bias)]

# Helper to compute the L2 norm of a vector (Euclidean distance from zero)
def l2_norm(v):
    return math.sqrt(sum(xi**2 for xi in v))

# The attack's objective function: we want to minimize this.
# It combines the size of the perturbation (delta) and a term that encourages misclassification.
def objective(delta, c):
    # Perturbed input
    x_pert = [original[i] + delta[i] for i in range(len(original))]
    # Get model scores
    z = scores(x_pert)
    # Score for the target class
    z_t = z[target]
    # Maximum score among all other classes
    z_max_other = max(z[i] for i in range(len(z)) if i != target)
    # f is positive if the model is correct, negative if it misclassifies
    # We want to make f as low as possible, so we include it in the objective.
    f = max(z_max_other - z_t, 0)  # if z_t is already the max, this is negative? Actually keep simple.
    # For simplicity, let's define f as (z_max_other - z_t + 1) to push for misclassification
    # The +1 ensures we push for a margin of 1.
    f = max(z_max_other - z_t + 1, 0)
    # Objective = ||delta||^2 + c * f
    return l2_norm(delta)**2 + c * f

# Gradient of the objective with respect to delta, computed numerically (finite differences)
def gradient(delta, c, h=0.0001):
    grad = []
    for i in range(len(delta)):
        delta_plus = delta[:]
        delta_plus[i] += h
        delta_minus = delta[:]
        delta_minus[i] -= h
        grad.append((objective(delta_plus, c) - objective(delta_minus, c)) / (2 * h))
    return grad

# The attack: gradient descent to find the delta that minimizes the objective
# Start with zero perturbation
delta = [0.0, 0.0, 0.0]
learning_rate = 0.1
c = 1.0  # balance parameter
iterations = 100

for iter in range(iterations):
    grad = gradient(delta, c)
    # Move delta in the opposite direction of the gradient
    delta = [delta[i] - learning_rate * grad[i] for i in range(len(delta))]
    # Optional: print progress every 20 iterations
    if iter % 20 == 0:
        x_pert = [original[i] + delta[i] for i in range(len(original))]
        z = scores(x_pert)
        pred = z.index(max(z))
        print(f"Iteration {iter}: perturbation {delta}, prediction {pred}, L2 norm {l2_norm(delta):.4f}")

# Final results
x_pert = [original[i] + delta[i] for i in range(len(original))]
z = scores(x_pert)
pred = z.index(max(z))
print(f"Final perturbation: {delta}")
print(f"Perturbed input: {x_pert}")
print(f"Original input: {original}")
print(f"Model prediction on original: {scores(original).index(max(scores(original)))}")
print(f"Model prediction on perturbed: {pred}")
print(f"Perturbation size (L2 norm): {l2_norm(delta):.4f}")
print(f"Success: {pred == target}")
```


## The same thing, with a framework

```python
import torch
import torch.nn.functional as F

# Assume we have a PyTorch model 'model' that takes input and returns logits
# Here we define a simple linear model for illustration
model = torch.nn.Linear(3, 3)  # 3 inputs, 3 outputs
# Load pretrained weights (not shown)

# Original input as a tensor (requires_grad=False)
x = torch.tensor([1.0, 2.0, 0.5], requires_grad=False)
# Target class (the class we want to fool the model into predicting)
target = 2

# Initialize perturbation delta as a tensor that requires gradients
delta = torch.zeros_like(x, requires_grad=True)
# Hyperparameters
c = 1.0
learning_rate = 0.1
iterations = 100

# Optimizer (we could use SGD, but here we manually do gradient descent)
optimizer = torch.optim.SGD([delta], lr=learning_rate)

for iter in range(iterations):
    optimizer.zero_grad()
    # Perturbed input
    x_pert = x + delta
    # Get logits
    logits = model(x_pert)
    # Compute the objective: minimize ||delta||^2 + c * f(x_pert)
    # f = max(logit_target - max_other, -kappa) with kappa=0
    z_t = logits[target]
    z_max_other = max([logits[i] for i in range(len(logits)) if i != target])
    f = max(z_max_other - z_t + 1, 0)  # we use +1 as a margin
    loss = torch.norm(delta)**2 + c * f
    loss.backward()
    optimizer.step()

# Final check
x_pert = x + delta
logits = model(x_pert)
pred = torch.argmax(logits).item()
print(f"Original prediction: {torch.argmax(model(x)).item()}")
print(f"Perturbed prediction: {pred}")
print(f"Perturbation (delta): {delta}")
print(f"L2 norm of perturbation: {torch.norm(delta).item()}")
```


## What the code does

The from_scratch code is a minimal implementation of the Carlini-Wagner attack on a tiny toy model. The model is a linear function: for an input vector x, it computes a score for each class as the dot product of a weight vector and x, plus a bias. The model predicts the class with the highest score. The attack starts with a zero perturbation (delta) and defines an objective that combines the size of the perturbation (the squared L2 norm) and a term that encourages the model to misclassify the perturbed input into the target class. The function f is defined so that it is zero when the model already predicts the target class with enough margin, and positive otherwise. The attack then uses gradient descent: it computes the gradient of the objective with respect to delta using numerical finite differences, then updates delta in the opposite direction of the gradient, scaled by a learning rate. After many iterations, the delta converges to a small perturbation that makes the model output the target class. The framework code does the same, but uses PyTorch's automatic differentiation to compute the gradient exactly, and uses an optimizer (SGD) to update delta. The key difference is that PyTorch handles the gradient computation and the optimization loop, but the underlying arithmetic is the same.


## Common misconceptions

**The attack works by adding random noise** — I thought the attack just adds random noise to the image, and if the model is wrong, it's an adversarial example.

Random noise is unlikely to fool a model, and even if it does, the change is usually large and visible. Carlini-Wagner systematically minimizes the noise by optimizing it with gradient descent, so it finds the smallest change, not any change. In fact: The attack uses optimization to find the minimal perturbation that flips the model's decision. It is a directed search, not random noise.

**If a defence defeats a simple attack, it is secure** — If my model resists a few random attacks, it's secure against all attacks.

Many defences only work against naive attacks. A sophisticated attack like Carlini-Wagner, which adapts to the defence, can break them. This is why the research community insists on testing against adaptive attacks. In fact: Defences must be evaluated against adaptive attacks — attacks that are aware of the defence and designed to bypass it. Otherwise the claim is unsupported.

**The perturbation is always the same size** — Adversarial examples all have the same amount of noise, just a fixed amount added to each pixel.

Carlini-Wagner finds a perturbation that is different for each input — it is tailored to push that specific input just across the decision boundary. The size varies from input to input. In fact: The perturbation is input-specific and minimal; it varies depending on where the input lies relative to the model's decision boundaries.

**The attack only works on images** — Carlini-Wagner is only for image classifiers; it doesn't apply to text or audio.

The optimization framework is general — it works on any input that can be represented as numbers and any model that produces a differentiable output. It has been extended to speech recognition, text classification, and even reinforcement learning agents. In fact: The core idea applies broadly: find the minimal change to any input that flips the model's decision, as long as you can compute or estimate gradients.

**The attack needs to know the model's weights** — It only works in white-box settings; I can't use it if I don't have the model internals.

Although original Carlini-Wagner is white-box, the idea of adaptive attacks extends to black-box settings by estimating gradients from queries (e.g., using finite differences). There are black-box attacks inspired by Carlini-Wagner that work with only query access. In fact: The optimization principle can be applied with estimated gradients, so attacks inspired by Carlini-Wagner can work even when the model's internals are hidden.


## In short

The Carlini-Wagner attack finds the smallest possible change to an input that fools a model into misclassifying it. It does this by setting up an optimization problem that balances two goals — make the model wrong, and keep the change small — and then solving it with gradient descent. It is the benchmark attack: any defence that claims robustness must be tested against it, or the claim is unsupported. It connects to adversarial examples, gradient obfuscation, and model extraction, and it applies far beyond images, to text, audio, and even agents.


## Where this is used

The Carlini-Wagner attack is used as a benchmark for evaluating defences in AI security. If a new defence claims to make a model resistant to adversarial examples, the honest test is to run the Carlini-Wagner attack against it. Many defences that appeared strong against simpler attacks (like fast gradient sign method) collapse when tested with Carlini-Wagner. It is the gold standard for 'adaptive attacks' — attacks that are tailored to a specific defence, often by computing gradients through the defence or estimating them. In practice, researchers and security engineers use it in several contexts:

- **Large language models (LLMs)**: Although originally designed for image classification, the principles of Carlini-Wagner have been adapted to NLP. For instance, adversarial attacks on text can involve changing a few words or characters, but Carlini-Wagner's optimization approach can find minimal changes that flip the model's output. In LLMs, this is relevant for attacks that cause the model to produce harmful content or ignore safety instructions. The attack's focus on minimal perturbation connects to the idea of 'jailbreaks' — prompts that look normal but cause the model to break its rules.

- **Small models on devices**: For speech recognition on phones or keyword spotters, adversarial examples can be played over the air to trigger wrong commands. Carlini-Wagner has been used to generate audio perturbations that are nearly inaudible but cause the model to transcribe an entirely different command. The optimization is run offline to craft the audio, which is then played to the device — this is a realistic threat.

- **Retrieval-augmented generation (RAG)**: In RAG, the model queries a database to retrieve relevant documents. An attacker might craft a document that, when retrieved, causes the model to produce a wrong answer. Carlini-Wagner-style optimization could be used to generate a document that is similar to a real one but contains a hidden trigger that misleads the LLM. This is an emerging area, but the concept of minimal modifications applies directly.

- **Agentic systems**: Agents that call tools and act in loops are vulnerable to adversarial instructions embedded in tool outputs. An attacker could craft a tool response that is minimally different from a benign one but causes the agent to take a malicious action. The Carlini-Wagner principle — find the smallest change that flips the agent's decision — applies here, though the search space is more complex (text rather than pixels).

- **Everyday applied ML**: In fraud detection, a fraudster wants to make a fraudulent transaction look benign. Carlini-Wagner can generate minimal modifications to the transaction features (amount, time, etc.) that cause the model to classify it as legitimate. Similarly, in spam filtering, an attacker can optimize an email to bypass the filter with minimal changes. In biometric recognition, liveness detection systems can be attacked with adversarial examples generated by similar optimization methods.

Developers working on security use libraries like AdverTorch or Foolbox that implement Carlini-Wagner attack. They write code like `attack = CarliniWagnerL2(model, targeted=False)` and then `adversarial_example = attack(image, label)`. What breaks when done badly is that the attack is computationally expensive — it requires many iterations of gradient descent, so it may be slow for real-time testing. Also, if the attack is used incorrectly (e.g., with wrong parameters), it might not find the minimal perturbation, giving a false sense of security. The key lesson is that any defence claim must be tested against adaptive attacks like this one; otherwise it is unsupported.


---

Read this on the web: https://basavapurushottam.com/learn-ai/carlini-wagner-attack
