# Gradient Checking

Verifying an analytic gradient against a finite-difference approximation, the standard sanity check before trusting a hand-written backward pass.

**Rests on:** Automatic Differentiation, Computational Graphs

*Mathematical Foundations · Calculus*


## Why learn this

Imagine you have spent hours calculating the directions for a steep climb, but you haven't verified that your map is accurate. If the map is wrong, you might walk off a cliff. In building models that learn from data, this verification step is called gradient checking. It is a sanity check that confirms the mathematical rules you wrote down actually match how the system behaves in practice. Without it, you risk training a model that seems to work but is secretly learning the wrong patterns, leading to silent failures that are hard to diagnose later.


## The intuition

Let's look at a concrete example. Suppose you want to know how steep a hill is at your current location. You stand at point A, where the height of the ground is 100 meters. To measure the steepness, you take a tiny step forward to point B, which is very close by, and measure the new height as 102 meters. The change in height is 2 meters over that tiny distance. This ratio gives you an estimate of the slope.

Now, imagine your calculator claims the slope is exactly 2 meters per unit step. You can check if it is right by doing this physical measurement yourself. If your calculated number matches the measured number, you know your calculator is working correctly for this spot. Gradient checking does exactly this: it takes a tiny 'step' in one of the parameters (a weight in the network), measures how much the error changes, and compares that result to the complex formula the computer used to calculate the slope automatically. If they agree closely, the formula is correct.


## An analogy

Think of checking your GPS speedometer by timing yourself over a known short distance with a stopwatch. Your GPS gives you an instant, calculated speed based on satellite data—this is like the automated calculation in a computer program. To verify it, you use a stopwatch and a marked track of exactly 100 meters. If the GPS says you are going 36 kilometers per hour (which is 100 meters in 10 seconds), and your stopwatch confirms it takes 10 seconds to cover the distance, you know the GPS is reliable. You do not need to understand the satellites to verify the speed; you just compare two independent measurements of the same thing: how fast you are moving. The GPS is fast but can sometimes drift slightly due to signal issues; the stopwatch is slower but fundamentally grounded in time and distance. Comparing them tells you when to trust the GPS.


## Definition

Gradient checking is a method to verify that the calculated rate of change used for learning matches the actual observed change by testing the system with small adjustments to one parameter at a time.


## How it connects

This concept relies on **Calculus**, specifically the definition of a derivative as a measure of how much a function changes when its input changes slightly. It also relates to **Automatic Differentiation**, which is the technique computers use to compute those derivatives exactly by applying the chain rule repeatedly, rather than estimating them with small steps.


## The mathematics

This concept rests on **calculus**, specifically the idea of a **derivative** — a measure of how steep a function is at a point. To understand gradient checking, you first need to understand a derivative. Picture a hill. At the top, the ground is flat — the slope is zero. Walk down a little, and the slope is negative (going down). Walk back up the other side, and the slope is positive. The derivative is just the number that tells you how steep that slope is at any single point: positive means uphill, negative means downhill, and the size of the number says how steep.

Now, how do you *measure* a slope in practice? You pick two points close together on the hill, measure their heights, and divide the difference in height by the difference in horizontal distance. That is the formula for slope: rise over run. The official derivative is what you get when you make that horizontal distance infinitely small. But in a computer, you cannot use infinitely small numbers; you can only use very small finite ones, like 0.0001. So the *numerical* derivative is an approximation: you nudge the input by a tiny amount, see how much the output changes, and divide. That is called a **finite difference**.

There are two common ways to do it: *forward difference* (nudge forward, compare) and *central difference* (nudge backward and forward, compare the two heights). Central difference is more accurate because it averages out curvature. Now, in machine learning, your network calculates a gradient (the *analytic* or *backprop* gradient) using a clever algorithm that gives the exact slope. But that algorithm is complex and easy to write wrong. Gradient checking simply compares the analytic gradient to the numerical one computed with finite differences. If they match (within a small tolerance), your analytic math is probably correct. If not, there is a bug.

The central difference approximation of the derivative of a scalar function f at point x is:

f prime of x is approximately equal to [f of x plus h minus f of x minus h] divided by 2h

Here, f of x means the height of the hill at position x. The symbol prime after f means we are talking about the slope (derivative) of f. The letter h stands for a very small number, called the step size. We add h to x to get a point slightly to the right, and subtract h from x to get a point slightly to the left. We find the height at both points, take the difference between those heights, and divide by the total distance between the two points (which is 2h).

For a network with multiple weights theta (a vector), the gradient is a vector, and we check each component theta-i separately. The letter i is just an index, like a label for which weight we are looking at. Theta-i means the i-th weight in the list of all weights.

The partial derivative of L with respect to theta-i is approximately equal to [L of theta plus h times e-i minus L of theta minus h times e-i] divided by 2h.

Here, L is the loss function, which tells us how bad the network's prediction is. Theta is the list of all weights. e-i is a special vector that has a 1 in position i and 0 everywhere else. Multiplying theta by e-i isolates the i-th weight. So theta plus h times e-i means we take the original list of weights, and add a tiny amount h only to the i-th weight, leaving all other weights unchanged. We then calculate the loss L for this new set of weights. We do the same thing but subtract h instead of adding it. Finally, we take the difference between these two losses and divide by 2h to get the approximate slope for just that one weight.


## A worked example

``` 
We have a simple function f of x equals x squared. We want to find the gradient (slope) at x equals 3. The true derivative is f prime of x equals 2x, so at x equals 3, the true slope is 6. But let us pretend we do not know that, and use numerical approximation to check a (supposed) analytic value. We will use a step size h equals 0.0001.

Step 1: Compute the point slightly to the left.
x minus equals x minus h = 3 minus 0.0001 = 2.9999

Step 2: Compute the point slightly to the right.
x plus equals x plus h = 3 plus 0.0001 = 3.0001

Step 3: Evaluate the function at the left point.
f of x minus equals f of 2.9999 = 2.9999 squared
To calculate 2.9999 squared, we multiply 2.9999 by 2.9999.
2.9999 * 2.9999 = 8.99940001
So, f of x minus equals 8.99940001.

Step 4: Evaluate the function at the right point.
f of x plus equals f of 3.0001 = 3.0001 squared
To calculate 3.0001 squared, we multiply 3.0001 by 3.0001.
3.0001 * 3.0001 = 9.00060001
So, f of x plus equals 9.00060001.

Step 5: Calculate the difference in heights.
difference in height = f of x plus minus f of x minus
difference in height = 9.00060001 minus 8.99940001
difference in height = 0.00120000

Step 6: Calculate the total horizontal distance.
total distance = 2h = 2 * 0.0001 = 0.0002

Step 7: Divide the difference in height by the total distance to get the numerical gradient.
numerical gradient = difference in height / total distance
numerical gradient = 0.00120000 / 0.0002
To divide, we can move the decimal point four places to the right for both numbers:
0.00120000 becomes 12
0.0002 becomes 2
12 divided by 2 equals 6.
So, numerical gradient = 6.000000000001 (due to floating point precision, it is extremely close to 6).

Result: The numerical gradient is approximately 6, which matches the true analytic gradient (2 times 3 equals 6). If the network's backprop gave us a value like 5.8, we would know there is a bug. The central difference gives a very accurate approximation because the two-sided average cancels out the error from the curve's curvature.
```


## From scratch, in Python

```python
# Gradient checking by hand, for a tiny network

# We'll use a simple function: f(x) = x^2
# and a simple 'analytic' gradient that we suspect might be wrong

def f(x):
    return x**2

def analytic_gradient(x):
    # We think this is correct, but we want to check
    return 2*x  # This is actually correct, but pretend we're not sure

# Step 1: choose a point to check
x = 3.0

# Step 2: choose a step size
h = 0.0001

# Step 3: compute the central difference
f_plus = f(x + h)
f_minus = f(x - h)
numeric_grad = (f_plus - f_minus) / (2*h)

# Step 4: compare
a = analytic_gradient(x)
b = numeric_grad

# Step 5: relative error
relative_error = abs(a - b) / max(abs(a), abs(b), 1e-8)  # tiny denominator guard

print(f"Analytic gradient: {a}")
print(f"Numeric gradient: {b}")
print(f"Relative error: {relative_error}")

if relative_error < 1e-7:
    print("Gradient check passed! Analytic gradient is correct.")
else:
    print("Gradient check FAILED. There's a bug in your analytic gradient.")

def check_layer_weights(weights, loss_fn, h=1e-5):
    """Check the gradient for a whole vector of weights."""
    n = len(weights)
    numeric_grad = [0.0] * n
    for i in range(n):
        # Make a copy of weights so we don't change the original
        w_plus = weights[:]
        w_minus = weights[:]
        w_plus[i] += h
        w_minus[i] -= h
        loss_plus = loss_fn(w_plus)
        loss_minus = loss_fn(w_minus)
        numeric_grad[i] = (loss_plus - loss_minus) / (2*h)
    return numeric_grad

# Example usage:
# Suppose we have weights = [1.0, -2.0, 0.5]
# and a loss function that computes (sum of squares of weights) / 2
# The analytic gradient is just the weights themselves.
# But let's pretend we wrote a buggy version: gradient = [2*w0, w1, w2]
# We'd run the check and see it fail.

weights = [1.0, -2.0, 0.5]

def loss(w):
    return sum(wi**2 for wi in w) / 2

def analytic_gradient_vector(w):
    return [2*w[0], w[1], w[2]]  # buggy: first weight is multiplied by 2

numeric = check_layer_weights(weights, loss)
analytic = analytic_gradient_vector(weights)

for i in range(len(weights)):
    err = abs(analytic[i] - numeric[i]) / max(abs(analytic[i]), abs(numeric[i]), 1e-8)
    print(f"Weight {i}: analytic={analytic[i]:.6f}, numeric={numeric[i]:.6f}, rel_err={err:.2e}")
```


## What the code does

First, we define two functions: `f(x)` computes the loss (in this case, x squared), and `analytic_gradient(x)` returns what we *think* is the derivative. We deliberately wrote it as `2*x`, which is the correct derivative, but later we'll see a buggy version. Next, we set the point we want to check (`x = 3.0`) and a small step size `h`. The `h` is a tiny number — too large and the approximation is inaccurate; too small and you get floating-point errors. Then we compute the central difference: we evaluate `f` at `x + h` and `x - h`, subtract the second from the first, and divide by `2*h`. This gives us a numerical approximation of the slope. The `relative_error` is the absolute difference between the analytic and numeric values, scaled by the size of the numbers being compared. This is what the library notes call 'relative rather than absolute error is the right comparison'. The reason is that an absolute difference of 0.0001 is significant if the gradient is tiny, but negligible if the gradient is huge. The relative error gives a scale-invariant measure. Finally, I included a helper function `check_layer_weights` that checks an entire vector of weights. It makes a list to hold the numeric gradients, then loops over each weight. For each weight `i`, it makes two copies of the weight list — one with `h` added to that weight, one with `h` subtracted. It computes the loss for both, and uses those to get the central difference for that weight. The loop runs from `0` to `n-1`, and `range(n)` generates those indices. Lists are Python's way of holding an ordered collection of items, and here we use them to store the weights. The `[:]` syntax makes a copy of the list so we don't modify the original. At the end, we compare each analytic gradient component to its numeric counterpart and print the relative error.


## Common misconceptions

**Gradient checking is only for beginners** — Once I've written one successful backprop, I don't need to check gradients again.

Gradient checking is a standard practice even for experienced developers, especially when writing custom layers or loss functions. It's a safety net that catches subtle bugs like an off-by-one index or a missing transpose. In fact: Gradient checking is a best practice whenever you implement a new backward pass. It's a quick test that saves hours of debugging later.

**Absolute error is fine** — If the numeric gradient and analytic gradient are both 0.5, they're close enough — they differ by 0.0001, which is small.

If the true gradient is 0.0001, an absolute error of 0.0001 is a 100% error, which is catastrophic. Absolute error doesn't tell you how significant the difference is. In fact: Use the relative error, which is the absolute difference divided by the magnitude of the two gradients. A relative error below 1e-7 is generally safe.

**Any step size works** — I'll use h = 0.01. That's small enough to be a good approximation.

If h is too large, the approximation is poor because the function curves within the step. If h is too small (e.g., 1e-12), round-off errors from floating-point arithmetic dominate. A good h is around 1e-5 for float64. In fact: Choose h carefully. Common values are 1e-4 to 1e-6. Test with different values; if the relative error changes dramatically, h is probably wrong.

**Passing the gradient check means everything is correct** — My gradient matched numerically, so my model must be perfect.

Gradient checking only verifies that the mathematical derivative is implemented correctly for a given set of inputs. It does not guarantee that the model architecture, loss function, or optimization process is sound. Also, it checks a few random points, not all possible inputs. In fact: Gradient checking is a necessary but not sufficient condition. It confirms your backward pass is implemented correctly, but you still need to validate your model on hold-out data and monitor training.

**The gradient is the loss** — Gradient checking is the same as checking the loss function numerically.

The loss is a scalar; the gradient is a vector of partial derivatives. You can evaluate the loss numerically, but gradient checking specifically verifies the derivative of that loss with respect to every parameter. In fact: Gradient checking compares the analytic gradient (from backprop) to the numeric gradient (from finite differences). It's a check on the derivative, not on the loss value itself.


## In short

Gradient checking is the sanity check you run before trusting any custom backward pass. You compute the analytic gradient (from backprop) and compare it to a numerical approximation (central difference) with a small step size. Use relative error, not absolute, and aim for less than 1e-7. It's a standard practice in every area of deep learning, from LLMs to edge devices, and it's the difference between debugging for days and moving on to model innovation.


## Where this is used

**In large language models (LLMs):** During development, when a research team writes a novel attention mechanism or a new loss function (like a custom contrastive loss), they use gradient checking to verify the backward pass. LLMs have billions of parameters, and a single wrong sign in a gradient can make training diverge. The developer writes the custom layer, then runs a gradient check on a small random subset of weights before kicking off a multi-million-dollar training run. For example, in a Transformer's self-attention, if the gradient for the query or key projections is wrong, the model may never learn to attend correctly. The developer would write the forward pass, then implement the backward pass by hand, and use gradient checking to validate it against a numerical approximation. This is especially critical because the chain rule for attention involves several matrices, and a mismatch in shapes or a missed term is easy to introduce. The library note from *Deep Learning From Scratch with Python* emphasizes this: 'Matching the numeric gradient is the standard sanity check for any backprop implementation — write it before you trust the network.' In practice, they would use a library like PyTorch, which has a built-in function for this: `torch.autograd.gradcheck`. Underneath, it does exactly the finite difference we wrote in our plain Python, but vectorised and parallelised. The developer would write a small unit test that calls `gradcheck` on their custom function, and it returns `True` only if the analytic gradient matches the numeric one to within a tight tolerance. If it returns `False`, they debug their backward pass, not the forward pass.

**In small language models running on phones or laptops (TinyML/edge):** When deploying a model to a device, you often need to quantise it — converting 32-bit floating-point numbers to 8-bit integers to save memory. The quantisation process can alter the gradients during fine-tuning. If the developer implements a custom quantisation-aware training layer, they use gradient checking to ensure the backward pass through the quantised weights is correct. They would write the forward pass that simulates quantisation (e.g., rounding weights to 8-bit) and a backward pass that approximates the gradient (often using a technique called the straight-through estimator). Because the derivative of the round function is zero everywhere, they replace it with a pass-through — this is an approximation, and gradient checking is used to confirm that the approximation is 'close enough' to the true numerical gradient (which would require the round function to be differentiable, which it isn't, so they check with a tolerance). A developer writes a script that instantiates a small CNN, computes the loss on a batch, calls `loss.backward()`, and then uses a numerical gradient checker on a few random weights to compare.

**In retrieval-augmented generation (RAG):** RAG systems use embeddings to retrieve relevant documents. The embedding model is trained with a contrastive loss (like InfoNCE), which pulls positive pairs together and pushes negatives apart. The gradient of this loss involves a softmax over a large batch, and it's easy to make an off-by-one error in the indices. Developers writing custom contrastive heads or loss functions use gradient checking to validate the backward pass for the similarity matrix computation. They write the forward pass that computes dot products between query and key embeddings, then the backward pass that computes gradients for each embedding. They then run a gradient check on a small batch (e.g., 4 examples) to ensure the gradients match numerically. In practice, they'd use a framework like PyTorch or TensorFlow, which provide autograd, but if they're implementing a custom loss function in pure NumPy for a production system (perhaps to reduce dependencies), they'd write a gradient checker from scratch, as we did above, and run it in CI (continuous integration) on every commit.

**In agentic systems that plan, call tools, and act in loops:** Agentic systems often use reinforcement learning or fine-tuning with a custom reward signal. The gradient of the reward with respect to the policy parameters is critical. If an agent's policy is a neural network, and it uses a custom loss (e.g., a combination of policy gradient and value loss), the developer must ensure the gradient is correct. For example, in Proximal Policy Optimization (PPO), there's a clipped surrogate objective; the clipping logic has a piecewise derivative, and getting the condition wrong is a common bug. A developer would write a small unit test that computes the loss for a fake batch of experiences, then uses gradient checking to validate the ratio of new policy probability to old policy probability. If the gradient is wrong, the agent will fail to improve even with correct rewards. The developer writes something like:
```python
import torch
policy = nn.Linear(4, 2)
optimizer = torch.optim.Adam(policy.parameters())
# compute loss, loss.backward(), then compare with numerical check
```
They might even use `torch.autograd.gradcheck` on the clipped objective function directly.

**In everyday applied machine learning (recommendation, vision, forecasting, fraud detection, speech):** A data scientist training a custom model — say, a recommender with a two-tower architecture, or a fraud detection model with a custom loss that penalises false negatives more — will often add a gradient check to their training script. They do this after implementing a new layer or loss. For example, in a forecasting model using a custom quantile loss, the gradient has a condition based on the sign of the error. The developer writes the loss and its gradient by hand, then runs a gradient check on a small batch (e.g., 10 samples) before training on millions. If the check fails, they fix the bug — it's far cheaper than discovering after hours of training that the model isn't converging. In practice, they'd use PyTorch's `torch.autograd.gradcheck`, which they can call like:
```python
input = torch.randn(10, dtype=torch.double, requires_grad=True)
torch.autograd.gradcheck(quantile_loss, input)
```
And if it raises an error, they know the implementation is wrong.


---

Read this on the web: https://basavapurushottam.com/learn-ai/gradient-checking
