# Activation Patching

Deliberately altering internal activations during a forward pass to test whether a component causally drives a behaviour.

**Rests on:** Forward Propagation

*Trust, Governance and Ethics · Model Interpretability*


## Why learn this

Imagine you are managing a large kitchen where dishes are prepared in stages. You want to know exactly which ingredient or cooking step is responsible for a specific flavor in the final dish. Without this method, you can only taste the finished meal and guess what went into it. With it, you can swap out an ingredient halfway through the process—like replacing sugar with salt—and see if the final taste changes. This allows you to move from guessing why something happened to proving it by changing one thing at a time and watching the result. It turns observation into experimentation, letting you verify your hypotheses about how the AI arrives at its answers.


## The intuition

To understand this, imagine a factory assembly line making cars. The car starts as a chassis, then gets wheels, then an engine, then paint, and finally leaves the factory finished. The 'forward pass' is simply the process of the car moving from station to station, getting built layer by layer.

Now, suppose you want to know if the engine installation (layer 3) is responsible for the car driving forward later on. You could stop the assembly line after the wheels are attached (the middle of the 'forward pass') and take that specific car. Then, instead of letting the factory attach the standard engine, you swap it with an engine from a different model—or even remove it entirely. This is the 'intervention'.

Let’s use numbers to make this concrete. Imagine each layer adds a number to a total score. Layer 1 adds 5. Layer 2 adds 10. The original process continues: Layer 3 would have added 20, and Layer 4 would add 50, for a final output of 85. 

To test if Layer 3 is important, you pause the car after Layer 2 (where the score is 15). You normally expect Layer 3 to add 20. Instead, you patch in a value of 0. The score stays at 15. You continue to Layer 4, which still adds 50. The final output is now 65, not 85. Because the final number changed from 85 to 65 after you altered the middle value, you know Layer 3 was causally responsible for that part of the result.

You can also try swapping in a random number. If Layer 3 normally adds 20, you patch in a random value like 7. The new total might be 72. If you patch it with 0 and the output doesn't change at all, then Layer 3 was not important for that specific outcome. This is how you test causality: you change one mid-process number and see if the final answer shifts.


## An analogy

Think of a relay race where four runners pass a baton to determine the winner. The team's final time depends on everyone running their leg correctly. You suspect the second runner, Bob, is crucial for the win. To test this without letting him run his actual lap, you stop the baton handoff after the first runner finishes. Instead of giving the baton to Bob, you give it to a dummy stand-in who runs zero meters (representing a 'patch' of zero activity). You let the third and fourth runners complete their laps. If the team’s final time is significantly worse than if Bob had run his normal lap, you have proven that Bob was causally necessary for their good result. The analogy breaks down slightly because in an AI, the 'baton' carries complex information rather than just physical motion, and there are billions of runners, not four. But the core logic holds: you intervene at a specific handoff point to see if that runner matters.


## Definition

Activation patching is a method where you pause an AI’s step-by-step calculation midway, replace the current internal value with a different number (such as zero or a value from another input), and then continue the rest of the calculation to see if the final answer changes.


## How it connects

This concept sits beside 'forward pass' — which is simply the sequence of steps an AI takes to turn an input into an output — and helps you understand what happens inside that sequence. It follows closely after 'feature attribution', a technique that highlights which parts of the input were most noticeable, by moving beyond noticing to actually testing cause and effect.


## The mathematics

Activation patching does not introduce new mathematics of its own; it uses the arithmetic of the forward pass, which you already know. The branch of mathematics at play is simple arithmetic — addition and multiplication of numbers. No calculus, no algebra beyond multiplying and adding. The core idea is that during a forward pass, every neuron computes a weighted sum of its inputs, adds a bias, and passes the result through a squashing function. That intermediate number, before the squashing, is called the raw score or pre-activation. After the squashing, it is the activation. Activation patching says: let me take that activation (a single number, say 0.83), and instead of letting the model use 0.83, I will force it to use a different number, say 0.2. Then I will continue the forward pass from there, and see what final output comes out. The mathematics is just replacing a number in a calculation and redoing the rest of the arithmetic. The skill is not in the calculation — it is in choosing which activation to replace and with what, and interpreting the result.

Let a_i be the original activation of neuron i in a given layer. Let a_i' be the patched (replacement) value. Let the rest of the network's computation be represented by a function f that takes the activation vector a (all the activations in that layer) and produces the final output. Then: f(a) = y_original, and f(a') = y_patched, where a' is the same as a except that the i-th component is replaced with a_i'. The causal effect of neuron i on this output is measured by the difference y_original − y_patched, or sometimes by the ratio (y_original − y_patched) / y_original, expressed as a percentage change.


## A worked example

```
Imagine a tiny network with two layers. The first layer has two neurons, and the second has one output neuron. We pass in an input and get an output. We will focus on the first neuron of the first layer. During the forward pass, it computes an activation of 0.83. The second layer takes that activation (along with the other neuron's activation, say 0.5) and computes the final output. We will patch the first neuron's activation to 0.2 and see how the output changes.

Step 1: Original: output = (0.83 × 2) + (0.5 × 1) + 0.1 = 1.66 + 0.5 + 0.1 = 2.26   # This is the normal forward pass. The second layer combines the two activations with weights 2 and 1, and a bias of 0.1.
Step 2: Patched: output = (0.2 × 2) + (0.5 × 1) + 0.1 = 0.4 + 0.5 + 0.1 = 1.0   # We replace the 0.83 with 0.2, and redo the arithmetic. The output drops from 2.26 to 1.0.
Step 3: Change in output = 2.26 − 1.0 = 1.26   # This large change suggests the first neuron is causally important for this output. If the output had barely changed, we would know it was not.

Result: The patched output is 1.0, versus the original 2.26, a drop of 1.26. This tells us the first neuron is a strong driver of the output for this input.
```


## From scratch, in Python

```python
import random

# A tiny neural network with 2 inputs, 1 hidden layer of 2 neurons, and 1 output neuron.
# The weights and biases are just made-up numbers for this example.

# Hidden layer weights: row for each hidden neuron, column for each input.
hidden_weights = [
    [0.5, -0.2],  # weights for hidden neuron 1
    [0.3, 0.8]    # weights for hidden neuron 2
]
hidden_biases = [0.1, -0.3]

# Output layer weights: one weight for each hidden neuron.
output_weights = [2.0, 1.0]
output_bias = 0.1

def forward_pass(inputs, patch_hidden1=None):
    # Compute hidden layer activations (using a simple identity activation, just for clarity)
    # This is like: weighted sum of inputs + bias
    hidden_1 = hidden_weights[0][0] * inputs[0] + hidden_weights[0][1] * inputs[1] + hidden_biases[0]
    hidden_2 = hidden_weights[1][0] * inputs[0] + hidden_weights[1][1] * inputs[1] + hidden_biases[1]

    # If patching, override the first hidden neuron's activation
    if patch_hidden1 is not None:
        hidden_1 = patch_hidden1

    # Compute output
    output = hidden_1 * output_weights[0] + hidden_2 * output_weights[1] + output_bias
    return output, hidden_1, hidden_2

# Example input
inputs = [1.0, 2.0]

# Original forward pass
orig_output, orig_h1, orig_h2 = forward_pass(inputs)
print("Original output:", orig_output)
print("Hidden 1 activation:", orig_h1)

# Patch the first hidden neuron to 0.0
patched_output, _, _ = forward_pass(inputs, patch_hidden1=0.0)
print("Patched output:", patched_output)

# Measure the change
change = orig_output - patched_output
print("Change in output:", change)
```


## The same thing, with a framework

```python
import torch
import torch.nn as nn

# Define a simple two-layer network
class TinyNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden = nn.Linear(2, 2)
        self.output = nn.Linear(2, 1)

    def forward(self, x):
        h = self.hidden(x)
        # To patch, we would hook into the forward pass here
        y = self.output(h)
        return y, h

model = TinyNet()
inputs = torch.tensor([1.0, 2.0])

def forward_with_patch(model, inputs, patch_hidden1=None):
    x = inputs
    # First layer
    h = model.hidden(x)
    # Patch: replace the first element of h
    if patch_hidden1 is not None:
        h[0, 0] = patch_hidden1
    y = model.output(h)
    return y, h

# Original
output_orig, h_orig = forward_with_patch(model, inputs)
# Patched
output_patched, h_patched = forward_with_patch(model, inputs, patch_hidden1=0.0)

print("Original output:", output_orig.item())
print("Patched output:", output_patched.item())
print("Change:", (output_orig - output_patched).item())

# In real libraries, the patch is often done with a forward hook.
# A forward hook is a function that runs when a layer's output is computed,
# and it can modify that output before it is passed on.
```


## What the code does

We start by defining a tiny network with hard-coded weights and biases. The function forward_pass takes an input and an optional patch_hidden1. Inside, we compute the first hidden neuron's activation as a weighted sum of the inputs plus its bias, and the same for the second. If a patch is provided, we override the first hidden neuron's value. Then we compute the output as a weighted sum of the hidden activations plus the output bias. We run it normally, print the output and the first hidden value, then run it again with the patch set to 0.0, and print the difference. This is exactly what activation patching does: change a number mid-culation and observe the result.


## Common misconceptions

**Patching is the same as feature attribution** — If I use a saliency map, I know which input pixels matter. Patching is just another way to draw a heatmap.

Feature attribution is correlational: it looks at which inputs are associated with the output. Patching is causal: it changes the internal state and observes the effect. Correlation does not imply causation — a feature can be highly correlated with an output without causing it. In fact: Activation patching is a causal intervention. You actively change an internal value and see if the output changes. That is a different and stronger kind of evidence.

**Patching is only for researchers** — I'm a practitioner, not a researcher. I would never need to patch activations.

Activation patching is used in debugging production models, pruning small models for devices, and validating that a RAG system actually uses context. It is a practical debugging tool. In fact: Developers use patching to understand why a model fails, to compress models, and to verify that a model is relying on the right information.

**If the output changes, that neuron is the cause** — If I patch a neuron and the output changes a lot, that neuron is definitely the cause of the behaviour.

The change might be an artifact — the network might rely on a distributed representation, and disrupting any single neuron causes a large change, even if that neuron is not specifically 'about' the behaviour. Also, the patched value might cause the network to go into a regime it was not trained for. In fact: A large change is evidence of causal involvement, but you need to test multiple patches, use clean 'counterfactual' values, and compare against baselines to be confident.

**Patching a neuron to zero is the same as removing it** — If I set a neuron's activation to zero, it's as good as deleting it from the network.

Setting to zero changes the value but the neuron still exists and still passes its output to the next layer. Removing it would also change the network's architecture. A zero might be a completely unnatural value for that layer, causing cascading effects. In fact: Zero is just one possible patch. A better patch is often to use the activation from a different input, which is a more 'natural' counterfactual.

**Activation patching requires access to the model's internals** — I can't use activation patching unless I have full access to the model's weights and activations.

Many modern models are accessed via APIs, but some allow gradient information or have open-source versions. Even with a black-box API, you can use techniques like 'activation patching via output probing' which uses gradients to estimate the effect. In fact: Full access makes patching easier, but it is not strictly required. There are methods to approximate causal effects without direct access.


## In short

Activation patching is a causal intervention: you change an internal activation of a neural network during the forward pass, and observe how the output changes. This tells you whether that component is causally responsible for a behaviour, not just correlated with it. It is the foundation of mechanistic interpretability and a key tool for AI security, debugging, and model trimming. Always remember: correlation is not causation; patching gives you causation.


## Where this is used

Activation patching is not just an academic exercise; it is a practical tool used across the AI industry.

**Large Language Models (LLMs):** LLMs like GPT-4 have billions of parameters, and it is often unclear why they produce a certain answer. Researchers use activation patching to isolate which layers or attention heads are responsible for, say, the model's ability to perform arithmetic, follow instructions, or express a certain bias. For example, by patching activations in the middle layers of the model with activations from a different input, they can see if the model 'changes its mind'. This is crucial for debugging and for building safer models. A developer might use a framework like TransformerLens (built on PyTorch) to hook into specific layers and patch them, running many experiments to map which components are causally relevant.

**Small Language Models (running on a phone or laptop):** When a model must run on a device, efficiency is paramount. Activation patching is used to prune — to find which neurons are not causally important for any task, and then remove them. If patching a neuron to zero never changes the output, that neuron is redundant and can be eliminated, making the model smaller and faster without losing accuracy. This is a form of model compression.

**Retrieval-Augmented Generation (RAG):** In a RAG system, the model retrieves relevant documents and then generates an answer. Activation patching can help you verify that the model is actually using the retrieved context. You can patch the activations in the attention layer that correspond to the retrieved document tokens and replace them with something unrelated. If the answer drastically changes, you know the model was causally depending on that context. This is used to debug retrieval failures and to ensure the model is not ignoring the retrieved information.

**Agentic Systems:** Agents that plan and call tools rely on a sequence of decisions. Activation patching can be used to verify which internal decision is driving the use of a particular tool. For example, if an agent decides to call a search engine, you could patch the activation of a neuron thought to represent 'needs to search' and replace it with the activation from an input where the agent did not search. If the tool call disappears, you have found the causal mechanism. This is also used in red-teaming to find vulnerabilities where a malicious prompt can trigger an unwanted tool call.

**Everyday Applied ML (recommendation, vision, fraud detection, speech):** In any classification task, activation patching can be used for debugging and for building trust. For a fraud detection model, patching the activation of a particular neuron could tell you whether that neuron is truly responsible for flagging a transaction as fraudulent. If you patch it and the prediction changes from 'fraud' to 'legitimate', you know that neuron is key. In vision, you could patch the activations of a layer that corresponds to 'cat' and see if the model still predicts 'cat' when presented with a dog image. In speech, you could verify that a specific neuron is responsible for detecting a particular phoneme. A developer implementing this will typically use a deep learning framework like PyTorch and write a forward hook to modify the activations, then run the inference multiple times with different patches.


---

Read this on the web: https://basavapurushottam.com/learn-ai/activation-patching
