# Automatic Differentiation

Computing exact derivatives by tracking elementary operations and applying the chain rule mechanically, distinct from both symbolic differentiation and finite differences.

**Rests on:** Computational Graphs

**Leads to:** Gradient Checking, Newton Method, Self-Attention from Scratch

*Mathematical Foundations · Calculus*


## Why learn this

Every modern AI system — every large language model, every recommender, every speech recogniser — is trained by a single mathematical routine: adjusting millions of numbers until the model gets better. That adjustment is driven by a quantity called the gradient, and computing that gradient efficiently is the whole job of automatic differentiation. Without it, training a model with even a thousand parameters would take years; with it, a model with a billion parameters trains in weeks. This page gives you the one idea that makes every training run you will ever read about possible: how a computer calculates a derivative exactly and quickly, even when the function is millions of operations long. You will use this knowledge directly when you later meet backpropagation in neural networks, when you read about optimisers like Adam or SGD, and when you debug why a model is not learning — because the first thing anyone checks is whether the gradient is correct.


## The intuition

You have probably heard of the derivative: it measures how steep a curve is at a point, how fast something changes. A car speeding up: the derivative of its position is its speed; the derivative of its speed is its acceleration. That is the idea — a derivative is a rate of change. But in machine learning, the functions we need to differentiate are not simple curves like a parabola. They are enormous chains of operations: multiply this by that, add this, apply a squashing function, multiply by another weight, add a bias, and so on, millions of times. The derivative of such a chain is still a single number that says 'if I nudge this input a little, how much does the output change?' — but you cannot write it down by hand.

Automatic differentiation is the mechanical procedure that a computer follows to compute that number exactly, by breaking the huge calculation into tiny pieces, differentiating each piece, and combining the results. It relies on a rule called the chain rule: if A depends on B, and B depends on C, then the rate of change of A with respect to C is the rate of change of A with respect to B multiplied by the rate of change of B with respect to C. This rule lets the computer stitch together the tiny pieces into a complete answer.

It is not an approximation — it gives the exact derivative, to the precision of the computer's arithmetic. It is not magic — it is a systematic application of this chain rule. And it is not optional — it is the engine that powers every training loop in every deep learning framework.


## An analogy

Imagine you run a small delivery service. You have a warehouse, and your delivery driver follows a route: start at the warehouse, drive to the depot, then to the customer. The total time of the trip depends on many factors: how long it takes to load the van, how fast the driver goes, how long the traffic lights are. You want to know: if I shave one minute off the loading time, how much sooner does the customer get their parcel? That is a derivative — the sensitivity of the total time to a change in one part. You could do an experiment: load one minute faster, measure the new total, subtract the old total. That is finite differences — it requires doing the whole trip twice, and it gives an approximate answer, because traffic changes, and the effect of a one-minute change is not exactly the same as a one-second change. Instead, you can reason: loading time affects departure time, departure time affects arrival at the depot, and arrival at the depot affects arrival at the customer. If you know how each step depends on the previous one — 'if I leave one minute later, I arrive at the depot one minute later; if I arrive at the depot one minute later, I arrive at the customer one minute later' — then you can chain these: a one-minute saving at loading becomes a one-minute saving at the end. That is the chain rule, and automatic differentiation is just doing this chaining systematically, but for every step in the delivery route. And here is the key trick: you do not have to re-run the whole trip for each possible change. You do one forward run — the driver does the route — and then you walk backwards from the customer to the warehouse, asking at each step 'if I had changed the previous step by a tiny amount, how much would this step have changed?' That backward walk gives you the derivative of the total time with respect to every single decision along the route in one go, exactly. The analogy stops working when the route has branches and loops — but the principle holds: one forward pass, one backward pass, and you have every gradient.


## Definition

Automatic differentiation is a computer-based technique for computing the exact derivative of a function by recording every elementary arithmetic operation it performs, then applying the chain rule in reverse order to propagate gradient information from the output back to each input, in a single backward sweep.


## How it connects

You have not met the derivative yet, so this page teaches it from scratch. You have not met a 'function' in the mathematical sense either, so that is also built from nothing. After this page, you will be ready for the concept of computational graphs, which is the formal way to describe the recording step; for gradient checking, which uses a different method — finite differences — to verify that an automatic differentiation implementation is correct; and you will be ready to see how automatic differentiation sits inside every training loop you will later study, because every optimiser you will meet (gradient descent, Adam, etc.) calls on this gradient. The library records that automatic differentiation is the infrastructure that makes modern optimisation methods usable at scale — this page will make that claim concrete.


## The mathematics

To understand automatic differentiation, we first need to agree on what a derivative is and how it behaves when functions are chained together.

**What is a derivative?**
Imagine you are driving a car. Your position changes as time passes. The speedometer shows your instantaneous speed. Mathematically, the derivative of your position with respect to time is that speed. It tells you: if I nudge time forward by a tiny amount, how much does my position change?

More generally, for any function (a rule that turns an input number into an output number), the derivative at a specific point is a single number. This number represents the sensitivity of the output to small changes in the input. If the derivative is 5, it means that for every tiny step you take in the input direction, the output moves 5 steps in its direction.

**The Chain Rule: Connecting Steps**
Automatic differentiation relies on a principle called the chain rule. Imagine your calculation is not one big jump, but a sequence of small steps. You start with an input `x`. You apply operation 1 to get result `a`. Then you apply operation 2 to `a` to get result `b`. Finally, you apply operation 3 to `b` to get the final output `y`.

We want to know: how does a tiny change in `x` affect the final `y`?

The chain rule says we multiply the sensitivities of each step along the path. 
1. First, find how sensitive `a` is to `x`. Let's call this number `da/dx` (read as "the derivative of a with respect to x").
2. Next, find how sensitive `b` is to `a`. Call this `db/da`.
3. Finally, find how sensitive `y` is to `b`. Call this `dy/db`.

The total sensitivity of `y` to `x` is the product of these three numbers:

dy/dx = (dy/db) * (db/da) * (da/dx)

This works because the effect propagates linearly through each step. If step 1 amplifies a change by 2, and step 2 amplifies it by 3, the total amplification is 2 times 3, which is 6.

**Automatic Differentiation: The Reverse Pass**
Manual calculation of these products can become messy for long chains. Automatic differentiation (specifically the "reverse mode" often used in machine learning) does this systematically in two phases:

1. **Forward Pass:** You run the function normally. You start with `x`, compute `a`, then `b`, then `y`. You save every intermediate result (`a`, `b`) and every local derivative (the sensitivity of each step's output to its input) along the way. For example, you save `da/dx`, `db/da`, and `dy/db`.

2. **Backward Pass:** You start at the end. The derivative of the final output `y` with respect to itself is always 1 (because if `y` changes by 1 unit, it changes by 1 unit). Then you move backward through the saved steps, multiplying by the local derivatives:
   - First, compute `dy/da`. This is `dy/db` multiplied by `db/da`.
   - Next, compute `dy/dx`. This is `dy/da` multiplied by `da/dx`.

By the time you reach the start, you have the total derivative `dy/dx`. This method is exact (it uses precise formulas for each step, not approximations) and efficient because one backward pass gives you the sensitivity of the final output to *every* input variable simultaneously.


## A worked example

Let us walk through a concrete example. We will compute the derivative of a simple function using the reverse mode of automatic differentiation.

**The Function:**
We have an input `x`. The calculation proceeds in three steps:
1. Multiply `x` by 2 to get `a`.
2. Add 3 to `a` to get `b`.
3. Square `b` to get the final output `y`.

Mathematically, this is: 
y = (2 * x + 3)^2

We will evaluate this at **x = 5**.

**Step 1: The Forward Pass**
We compute the values of each variable in order.

- Input: `x = 5`
- Step 1: `a = 2 * x` 
  Calculation: `a = 2 * 5 = 10`
- Step 2: `b = a + 3` 
  Calculation: `b = 10 + 3 = 13`
- Step 3: `y = b * b` (squaring `b`)
  Calculation: `y = 13 * 13 = 169`

So, when the input is 5, the output is 169.

**Step 2: Save Local Derivatives**
Before we go backward, we must record how sensitive each step's output was to its immediate input. These are the "local derivatives."

- For Step 3 (`y = b^2`): The derivative of `b^2` with respect to `b` is `2 * b`. 
  At our current value `b = 13`, this local derivative is `2 * 13 = 26`.
  We save: `dy/db = 26`.

- For Step 2 (`b = a + 3`): The derivative of `a + 3` with respect to `a` is just 1 (adding a constant does not change the slope).
  We save: `db/da = 1`.

- For Step 1 (`a = 2 * x`): The derivative of `2 * x` with respect to `x` is just 2.
  We save: `da/dx = 2`.

**Step 3: The Backward Pass**
Now we compute the total derivative `dy/dx` by moving backward from `y` to `x`, multiplying by the saved local derivatives.

- Start at the output `y`. The derivative of `y` with respect to itself is always 1.
  `dy/dy = 1`

- Move back to `b`. We want `dy/db`. We take the derivative from above (`dy/dy`) and multiply by the local derivative we saved for step 3 (`dy/db`).
  `dy/db = (dy/dy) * (dy/db)`
  `dy/db = 1 * 26 = 26`
  This means if `b` changes slightly, `y` changes 26 times as much.

- Move back to `a`. We want `dy/da`. We take the derivative from above (`dy/db`) and multiply by the local derivative we saved for step 2 (`db/da`).
  `dy/da = (dy/db) * (db/da)`
  `dy/da = 26 * 1 = 26`
  This means if `a` changes slightly, `y` changes 26 times as much.

- Move back to `x`. We want `dy/dx`. We take the derivative from above (`dy/da`) and multiply by the local derivative we saved for step 1 (`da/dx`).
  `dy/dx = (dy/da) * (da/dx)`
  `dy/dx = 26 * 2 = 52`

**Result:**
The derivative of `y` with respect to `x` is **52**.

This number tells us that at the point where `x = 5`, if we increase `x` by a tiny amount (for example, 0.001), the output `y` will increase by approximately `52 * 0.001 = 0.052`. The calculation is exact because we used the precise formulas for the derivatives of multiplication and addition, not an approximation.


## From scratch, in Python

```python
# We'll start with a very simple function: y = (2x + 3)^2
# We'll compute the gradient of y with respect to x using automatic differentiation.

# Step 0: We need a function that records what operations it does.
# We'll create a tiny 'Value' class that stores a number and a gradient.

class Value:
    def __init__(self, data, _children=(), _op=''):
        self.data = data       # the number
        self.grad = 0.0        # gradient of the output with respect to this value, initially 0
        self._backward = lambda: None  # a placeholder function; we'll set it later

    # We'll implement the operations we need: __add__ for +, __mul__ for *, and so on.

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other), '+')

        # Define the backward function: how to propagate gradients through this operation.
        def _backward():
            self.grad += out.grad   # derivative of out with respect to self is 1
            other.grad += out.grad  # derivative of out with respect to other is 1
        out._backward = _backward
        return out

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other), '*')

        def _backward():
            # derivative of out with respect to self is other.data
            self.grad += other.data * out.grad
            # derivative of out with respect to other is self.data
            other.grad += self.data * out.grad
        out._backward = _backward
        return out

    # We also need a way to calculate the power, e.g., square.
    def __pow__(self, other):
        # only supporting integer powers for simplicity
        assert isinstance(other, (int, float))
        out = Value(self.data ** other, (self,), f'**{other}')

        def _backward():
            self.grad += (other * self.data ** (other - 1)) * out.grad
        out._backward = _backward
        return out

    def __repr__(self):
        return f"Value(data={self.data}, grad={self.grad})"

# Now we build the expression: y = (2*x + 3)^2
# We set x = 5
x = Value(5)

a = x * 2     # a = 2x
a = a + 3     # a = 2x + 3 (we reused the name for clarity)
y = a ** 2    # y = (2x + 3)^2

# Now we run the backward pass.
# We start by setting y's gradient to 1 (the derivative of y with respect to itself).
y.grad = 1.0

# We need to apply the backward in reverse topological order. 
# For simplicity, we'll just manually call them in the right order here.
# In a real system, we would sort the nodes.

# Backward for y = a**2
y._backward()

# Backward for a = a + 3 (the + node)
# But note: a was overwritten, so we need to keep a reference to the original nodes. 
# To keep this simple, we'll manually recompute the chain for the simple example.
# In practice, we'd store the graph.

# Let's start over with a cleaner implementation for this simple case.

# To avoid confusion, we'll build a small graph manually.
# Define a function that returns the value and its gradient.

# Let's implement a simple reverse-mode AD for a single variable case:

def forward(x):
    a = 2 * x
    b = a + 3
    y = b * b
    return a, b, y

def backward(x, a, b, y):
    # gradients of y w.r.t. a, b, x
    dydy = 1
    dydb = 2 * b  # derivative of b^2
    dbda = 1      # derivative of a+3
    dadx = 2      # derivative of 2*x
    # chain rule
    dyda = dydb * dbda
    dydx = dyda * dadx
    return dydx

x = 5.0
a, b, y = forward(x)
grad = backward(x, a, b, y)
print(f"x = {x}, y = {y}, gradient = {grad}")  # Expected output: gradient = 52
```


## The same thing, with a framework

```python
import torch

# PyTorch's autograd is exactly this backward pass, but generalised to huge graphs.
# You create tensors with requires_grad=True, and every operation builds the graph.

x = torch.tensor(5.0, requires_grad=True)
y = (2 * x + 3) ** 2
y.backward()  # this runs the backward pass, like our manual one
print(x.grad)  # tensor(52.)

# Under the hood, torch does exactly what we did: it records the operations (the forward graph),
# then walks backwards, using the local derivatives to accumulate the gradient.
```


## What the code does

The from-scratch code first defines a tiny Value class that holds a number and a gradient. When we add or multiply two Values, we create a new Value that remembers the operation and also defines a '_backward' function: for addition, we add the gradient of the output to both inputs; for multiplication, we multiply the gradient by the other input's value. To compute the gradient, we set the output's gradient to 1, then call the backward functions in reverse order (the simplest version does it manually, but the correct implementation would topologically sort the nodes). The more straightforward part of the code — our forward and backward functions — shows the steps with explicit formulas: compute the intermediate values, then apply the chain rule in reverse. In the PyTorch version, all that administrative work is hidden, but it is exactly the same arithmetic; the tensor with requires_grad=True records every operation, and calling backward() traverses the graph from output to inputs, accumulating the gradient.


## Common misconceptions

**AD is an approximation, like finite differences** — The computer cannot compute the exact derivative, so it must be estimating it with something like a slope between two points.

Finite differences do approximate, but automatic differentiation uses the exact derivative formulas for each elementary operation (constant, add, multiply, power, etc.) and combines them with the chain rule, which is an exact theorem. The only error is floating-point roundoff, unavoidable in any computation, but the result is as exact as the arithmetic itself. In fact: Automatic differentiation gives the exact derivative value (to machine precision), not an approximation. It is as exact as computing the forward function itself.

**AD is the same as symbolic differentiation** — Symbolic differentiation (like in a CAS that gives you a formula) and automatic differentiation are basically the same thing, because both use the chain rule.

Symbolic differentiation manipulates mathematical expressions to produce a new expression for the derivative, which can explode in size. Automatic differentiation works on numbers, not expressions: it evaluates the derivative at a specific point by accumulating products of local derivatives, and it never constructs a symbolic formula. It is closer to a numeric method, but exact. In fact: AD is a numerical method that computes a value for the derivative at a specific input, using the chain rule on the numeric values of intermediate results. It does not build a formula; it builds a computational path through numbers.

**Backpropagation is a separate algorithm from AD** — Backpropagation in a neural network is a different thing that I have to learn separately; automatic differentiation is just for maths functions.

Backpropagation is precisely the reverse-mode automatic differentiation of the network's loss function. The name is historical, but the algorithm is the same. When you later hear about backpropagation, you will simply be seeing AD at scale. In fact: Backpropagation is the application of automatic differentiation to a neural network's loss function. There is no extra mathematical step; it is the same backward pass we have practised here.

**One backward pass gives the gradient for just one input** — To get the gradient for each of a million weights, I need to run the backward pass a million times.

Reverse mode AD computes the derivative of the output with respect to every input in a single backward sweep, by propagating a single scalar gradient backward through the graph and accumulating at each node. This is the crucial efficiency that makes training large models feasible. In fact: A single backward pass computes the gradient with respect to all parameters at once, by accumulating the incoming gradient at each node. That is why it is so powerful.

**AD requires the function to be nice and smooth** — Automatic differentiation only works for simple polynomials or well-behaved functions; real-world machine learning functions are too messy.

AD works on any function that is composed of elementary operations (+, *, division, powers, exp, log, trigonometric functions, and so on). ReLU, for instance, is not differentiable at exactly zero, but it is differentiable almost everywhere, and AD handles the non-differentiable points by returning a sub-gradient (usually 0 or 1). Modern networks use lots of non-smooth functions, and AD handles them without issue. In fact: AD applies to any function made of elementary operations, including non-smooth ones like ReLU; at non-differentiable points, it simply returns a chosen sub-gradient.


## In short

Automatic differentiation computes the exact derivative of any function built from elementary operations. It works by recording the forward computation, then applying the chain rule in reverse. One backward pass gives the gradient with respect to every input. It is the engine behind backpropagation and every machine learning training loop. When you use a framework like PyTorch, `loss.backward()` is this idea doing its job. The two numbers to remember: a forward pass and a backward pass — that is all it takes.


## Where this is used

### Large Language Models
Every large language model, from GPT to Llama, is trained by adjusting millions of weights using a gradient computed by automatic differentiation. The model is a gigantic function: it takes a sequence of tokens, passes them through dozens of layers of matrix multiplications and attention computations, and produces a probability distribution over the next token. The 'loss' — how wrong the model's prediction is — is the final output. To improve the model, every weight needs to know how much a small change in it would reduce the loss, and that is exactly the gradient of the loss with respect to that weight. Training is: forward pass, compute the loss, then call .backward() — the automatic differentiation engine — which walks backwards through the entire network, using the chain rule, to fill each weight's .grad attribute. Then the optimiser steps: each weight subtracted by a small multiple of its gradient. This is repeated for trillions of tokens and billions of parameters. The efficiency of reverse mode — one forward and one backward pass gives all gradients — is precisely what makes this feasible at all. When you hear that a model was trained on thousands of GPUs, a large part of that computation is the backward pass, and any inefficiency there would multiply the cost astronomically.

### Small Language Models (on-device)
Small models that run on phones or laptops, like a keyword-spotting model or a small text autocomplete, are trained differently, but the training loop still uses automatic differentiation. The difference is scale: they might have a few million parameters instead of billions, but the gradient computation is still the core of the training loop. What changes at the edge is inference — running the model forward only, without backward. But when such a model is tuned on device (rare but possible for personalisation), automatic differentiation is used to update the weights from new user data, again through .backward().

### Retrieval-Augmented Generation (RAG)
RAG systems use embeddings to retrieve relevant documents. The embedding model — which converts text to a vector of numbers — is trained with automatic differentiation to make related texts have similar vectors and unrelated texts different vectors. The training objective is called a contrastive loss: it encourages the embedding of a question to be close to the embedding of its answer, far from the embeddings of random texts. The gradient of this loss with respect to the embedding model's weights is computed by automatic differentiation, and the model improves accordingly. While the retrieval step itself (the search) does not use gradients, the quality of the embeddings — which is what makes retrieval work — is entirely a product of gradient-based training that depends on automatic differentiation.

### Agentic Systems (Plan, Call Tools, Act)
Agentic systems that use an LLM to decide which tool to call, and then act on the result, typically use a language model that is trained with automatic differentiation. The language model's weights are updated through the same backward pass, whether the model is fine-tuned for tool-calling or trained from scratch. When the agent's actions are evaluated — say, it wrote a good or bad answer — the reward is converted into a gradient signal (through reinforcement learning or fine-tuning) that is propagated back through the model using automatic differentiation. So even though the planning loop (choose tool, call tool, observe result) is a hand-written program, the underlying model that makes the decisions is optimised by gradients, and automatic differentiation is the engine of that optimisation.

### Everyday Applied Machine Learning
Across the industry — recommendation systems, fraud detection, speech recognition, forecasting, computer vision — almost every model is a differentiable function, and almost every training script contains a line like `loss.backward()`. Recommendation models embed user and item IDs, predict a rating, and measure the error; the gradient of that error with respect to all embedding vectors is computed by automatic differentiation, and the embeddings are adjusted to make better recommendations. Fraud detection models learn a decision boundary from historical transactions, again through gradient descent. Speech recognition uses sequence models (something you will meet later) that are trained with a variant of backpropagation. In every case, the developer writes the forward pass — how the model turns input into output — and calls a backward method. The framework's autograd handles the derivative bookkeeping. The practical reality is that virtually nobody hand-derives a gradient for a modern model; they write the loss, call `.backward()`, and let automatic differentiation do the chain rule. The main practical issues are: memory (the backward pass stores intermediate activations, so large models can exhaust GPU memory), numerical stability (if a value becomes too large or too small, gradients can overflow or vanish), and debugging (if the model does not learn, the first question is often 'is my loss function differentiable?', and because automatic differentiation is exact, any error is almost certainly in the forward computation, not in the derivative).


---

Read this on the web: https://basavapurushottam.com/learn-ai/automatic-differentiation
