# Computational Graphs

Representing a calculation as a directed graph of operations, so the forward pass evaluates it and the backward pass traverses it in reverse to accumulate gradients.

**Rests on:** Tensors and Shapes

**Leads to:** Automatic Differentiation, Encoder-Decoder, Gradient Checking, Gradient Descent, Keras and TensorFlow, Shortcut Connections, Softmax and Logits, Training Dynamics, Upstream Gradients

*Mathematical Foundations · Calculus*


## Why learn this

This explains how a computer figures out which part of a complex calculation caused an error, so it can fix it. Imagine you are baking a cake, and it turns out too dry. You need to know if the oven was too hot or if you added too little milk. A computational graph is a map of every step in your process — from mixing flour to pouring batter to checking the temperature. By looking at this map, the computer can trace the 'dryness' back through each step to find exactly which ingredient or setting changed how much it matters. Without this map, the computer would have to guess and re-bake the cake thousands of times just to find a small fix. With it, you get an exact answer in one go. This is why your software can learn from its mistakes so quickly: it isn't guessing; it is reading the recipe backward.


## The intuition

Let us look at a specific example. Suppose you are calculating the total cost of buying apples and oranges. The steps are simple:
1. You buy 3 apples, and each apple costs $2. (Step 1: Multiply quantity by price for apples). Result: $6.
2. You buy 4 oranges, and each orange costs $1.50. (Step 2: Multiply quantity by price for oranges). Result: $6.
3. You add the cost of apples to the cost of oranges. (Step 3: Add the two results together). Final Total: $12.

Now, imagine the final total is wrong. You want to know how much changing the price of an apple would change that final $12 total. 

In a computational graph, this looks like three boxes connected by arrows. Box 1 outputs $6 into Box 3. Box 2 outputs $6 into Box 3. Box 3 outputs $12.

To find the 'importance' of the apple price to the final total, you start at the end ($12) and walk backward:
- At Box 3 (the addition), you ask: 'If I change the money coming from Box 1 by a tiny bit, how much does my total change?' Since you are just adding them together, if the apple cost goes up by $0.01, the total goes up by $0.01. So the link from Box 1 to Box 3 has a 'strength' of 1.
- Then you look at Box 1 (the multiplication). You ask: 'If I change the price of one apple by $0.01, how much does Box 1’s output ($6) change?' Since you bought 3 apples, changing the single-unit price by $0.01 changes the total apple cost by $3 * $0.01 = $0.03. So the link from the apple-price input to Box 1 has a 'strength' of 3.
- To find the total effect, you multiply the strengths along the path: 1 * 3 = 3. This means for every $0.01 increase in the apple price, the final total increases by $0.03.

You did this by walking backward through the steps you already took, using the numbers from those steps to figure out how sensitive each step was to its own input. You never had to re-do the whole calculation; you just traced the influence.


## An analogy

Think of a computational graph as a system of pipes carrying water. The forward pass is turning on the tap: water flows from the source, through various valves (the steps), and fills a tank at the end (the final answer). Each valve controls how much water passes through based on its setting.

The backward pass is like checking which valves are most responsible for the current water level in the tank. You start at the tank and look upstream. If you tighten one valve slightly, how much less water reaches the tank? You check each pipe connected to the tank, then follow those pipes back to where they came from. Because you know exactly how much water is currently flowing through each pipe (from the forward pass), you can easily calculate how a tiny adjustment at any point will ripple backward to affect the final level. The graph is the map of these pipes, ensuring you don't miss any connections or double-count flows where pipes split and merge.
Caveat: unlike real water, changing a parameter in one part of the calculation does not physically change the previous numbers; it only changes your understanding of how much they *would* matter.


## Definition

A computational graph is a diagram that breaks a calculation into individual steps, linking each step to the next with arrows. It allows you to work backward from the final result to determine exactly how sensitive that result is to every single input used in the process.


## How it connects

This concept sits beside 'Derivatives', which is just a formal way of describing how sensitive one number is to changes in another — essentially, the 'strength' or 'slope' of influence between two connected steps. It also connects directly to 'Automatic Differentiation', which is the method computers use to automatically trace these backward paths through any complex calculation without you having to do the algebra yourself.


## The mathematics

This concept rests on calculus, specifically on the notion of a derivative and the chain rule. You do not yet know what a derivative is, so we will build it from a simpler idea: the ratio of a small change in output to a small change in input.

Suppose you have a machine that takes a number in and gives a number out. If you increase the input by a small amount, the output changes by some amount. The ratio of that output change to the input change is a measure of how sensitive the machine is at that input value. That sensitivity is the derivative.

The chain rule says: if you have two machines in a row — first machine turns x into y, second turns y into z — then a small change in x produces a small change in y, and that produces a small change in z. The overall sensitivity of z to x is the product of the sensitivity of z to y and the sensitivity of y to x.

A computational graph uses this by representing a large calculation as a series of single operations, each with its own sensitivity. To find the sensitivity of the final output to any earlier input, you multiply the sensitivities along every path from that input to the output, and add the contributions of all paths — the chain rule again, applied repeatedly. That is the backward pass: exactly this multiplication and addition, done in reverse order from the output back to the inputs. The mathematical branch is calculus because the concept of a small change and its ratio is the foundation; the graph is the organisational structure that makes calculus practical for a network with millions of operations.

For a graph with nodes v1, v2, ..., vn, where the final output is vn, the backward pass computes, for each node i, the gradient ∂vn/∂vi. Starting from the output where ∂vn/∂vn = 1, for each node i in reverse topological order, apply: ∂vn/∂vi = sum over all outgoing edges from i to j of (∂vn/∂vj) * (∂vj/∂vi). Here, ∂vn/∂vj is the gradient already computed for node j, and ∂vj/∂vi is the local derivative of the operation at node j with respect to its input from i.


## A worked example

Consider a simple calculation with two operations. Start with input x = 3. Operation 1 multiplies x by 2 to give y = 2 * x. Operation 2 squares y to give z = y * y. We will compute the forward pass and then the backward pass to find how much z changes when x changes by a small amount.

Step 1: y = 2 * x = 2 * 3 = 6   # Forward pass, first operation: multiply the input by 2.
Step 2: z = y * y = 6 * 6 = 36   # Forward pass, second operation: square the intermediate value.
Step 3: sensitivity of z to y: if y increases by a small amount, say 0.1, then z goes from 6*6=36 to 6.1*6.1=37.21. Change in z is 1.21, change in y is 0.1, ratio is 12.1, approximately 12.   # Backward pass, starting from the output. The derivative of z with respect to y is 2*y = 12, which matches the numerical ratio.
Step 4: sensitivity of y to x: if x increases by 0.1, then y goes from 6 to 6.2. Change in y is 0.2, change in x is 0.1, ratio is 2.   # Backward pass, moving one step back. The derivative of y with respect to x is 2.
Step 5: overall sensitivity of z to x = (sensitivity of z to y) * (sensitivity of y to x) = 12 * 2 = 24   # The chain rule: multiply the sensitivities along the path from x to z.

Result: The gradient of z with respect to x is 24. This means if you increase x by a small amount, z increases by about 24 times that amount. The backward pass found this by walking from z back to x, multiplying local sensitivities.


## From scratch, in Python

```python
# Plain Python: implement the forward and backward pass of a tiny graph by hand.
# The graph: x -> (*2) -> y -> (square) -> z

# Forward pass
def forward(x):
    y = 2 * x          # first operation
    z = y * y          # second operation
    return z, y        # return output and intermediate for backward

# Backward pass: given the output gradient (start with 1),
# walk backwards multiplying local derivatives.
def backward(x):
    z, y = forward(x)
    dz_dz = 1.0        # gradient of z w.r.t. itself
    dz_dy = dz_dz * 2 * y   # local derivative of square: 2*y
    dz_dx = dz_dy * 2       # local derivative of *2: 2
    return dz_dx

# Test with x = 3
x_value = 3.0
z_value, y_value = forward(x_value)
grad_x = backward(x_value)
print("Forward: y =", y_value, "z =", z_value)
print("Backward: gradient dz/dx =", grad_x)  # should be 24.0
```


## The same thing, with a framework

```python
import torch

# Build the same graph using PyTorch tensors and autograd.
x = torch.tensor(3.0, requires_grad=True)
y = 2 * x
z = y * y
z.backward()   # walks the graph backwards automatically
print(x.grad)  # prints tensor(24.)

# Underneath, PyTorch builds a graph of operations when you write `y = 2 * x`.
# Each tensor remembers the operation that created it and its inputs.
# `z.backward()` walks that graph in reverse, exactly like the `backward`
# function above, computing gradients and storing them in `.grad`.
```


## What the code does

The code first defines `forward`, which computes the two operations in order and returns both the final output and the intermediate `y`. The intermediate is stored because the backward pass needs it: the local derivative of the square operation is `2 * y`, and `y` is not available unless we kept it. Then `backward` starts with a gradient of 1 for the output itself (the sensitivity of z to z is 1). It multiplies by the local derivative of the square, `2 * y`, to get the gradient with respect to y, then by the local derivative of the multiply-by-2 operation (which is 2) to get the gradient with respect to x. The loop is just two multiplications here, but the pattern generalises: every node stores its input value during forward, and then during backward it receives a gradient from the nodes that used its output, multiplies by its own local derivative, and passes the result to the nodes that fed it.


## Common misconceptions

**The graph is only for understanding, not for computation** — A computational graph is a nice diagram but the actual calculation just happens step by step.

The graph is the computation. The forward pass is the graph being evaluated, and the backward pass is the graph being traversed. The graph determines which values are stored and which derivatives are multiplied. In fact: The graph is an actual data structure built and used by the framework. Every operation creates a node, and the backward pass follows the edges.

**Backward pass requires recomputing the whole forward pass** — To get the gradient, the computer has to redo the entire forward calculation backwards from scratch.

The forward pass stored every intermediate value, so the backward pass only needs those stored values and the local derivatives — it does not recompute the forward calculations. In fact: The backward pass uses stored intermediates from the forward pass, which is why memory is the cost, not extra computation.

**The gradient is a single number for the whole model** — Backprop finds one number saying how wrong the model is.

The gradient is a separate number for every weight in the model — a vector of millions of values. In fact: The backward pass accumulates a gradient for every parameter in the graph, each telling you how that parameter should change to reduce the loss.

**A graph is a fixed diagram drawn before the code runs** — The model's architecture is a fixed picture, and the graph is just that drawing.

In 'define-by-run' frameworks like PyTorch, the graph is built as the Python code executes — different inputs can create different graphs if the code has branches or loops. In fact: The graph is constructed dynamically during the forward pass, which is what makes debugging feel like ordinary programming, as your notes record.


## In short

A computational graph is a recipe: nodes are operations, edges are values. The forward pass cooks and writes down every intermediate. The backward pass tastes the result and walks the written notes in reverse, multiplying local sensitivities (chain rule) to find how much each ingredient matters. This is backpropagation, and it is the engine of every training loop. Store your intermediates — they are what make the backward walk possible.


## Where this is used

Computational graphs are the skeleton of every deep learning framework — PyTorch, TensorFlow, JAX — and every model you train on them is a computational graph. In large language models, the forward pass through the transformer is a graph with billions of operations: embeddings, attention, feed-forward layers, softmax. During training, the loss is the final node, and backpropagation walks the entire graph in reverse, computing the gradient of the loss with respect to every weight. The library note 'Stored intermediate activations are the memory cost of backpropagation' is the practical bottleneck: every intermediate tensor from the forward pass is kept in memory so the backward pass can use it, which is why a 70-billion-parameter model needs hundreds of gigabytes of memory during training. In small models running on phones or laptops, the same graph is smaller, but the principle is identical — the model is a graph, and training it requires the backward walk. In retrieval-augmented generation, the retrieval step is not a neural network, but the generation model that reads the retrieved documents is a graph, and fine-tuning it for better retrieval-based answers again uses backpropagation through that graph. Agentic systems that plan, call tools, and act in loops typically do not train on every action — they use a pre-trained graph for each decision, but when they are fine-tuned, the exact same backward walk through the computational graph is what updates the policy. In everyday applied machine learning — recommendation systems, vision models, fraud detection, forecasting — the pattern is identical: build a graph of operations from input to prediction, compute a loss, and backpropagate. A developer working with PyTorch writes: `loss = loss_fn(model(x), y); loss.backward(); optimizer.step()`. The first line builds the graph, the second walks it backwards to compute gradients, and the third updates the weights. When it is done badly — for instance, when a custom operation is written without a correct backward function — the gradient is wrong, but often not obviously so; the model trains, but slowly or not at all. That is why the library notes 'Gradient Checking' as a sanity check: compare your analytic gradient to a finite-difference approximation before trusting a hand-written backward pass. The computational graph is not an abstraction you can ignore; it is the reason `backward()` works, and the reason memory usage scales with the number of intermediate activations.


---

Read this on the web: https://basavapurushottam.com/learn-ai/computational-graphs
