# Width vs Depth

Whether to add units per layer or add layers — a tradeoff between representational efficiency and optimisation difficulty.

**Rests on:** Forward Propagation

*Deep Learning · Deep Learning*


## Why learn this

This concept is the fork in the road for building any neural network. When you design a network, you must decide: do I make each layer wide (many neurons per layer) or do I stack many layers (deep)? This choice determines how well your model can learn, how much it costs to train, and whether it will fail to learn at all. Without this, you will build networks that are either too weak or impossible to train. This unlocks understanding of why modern architectures (like Transformers in LLMs, ResNets in vision) are shaped the way they are — they all balance width and depth. It is the first step toward designing your own models rather than copying others.


## The intuition

Think of a neural network as a team of workers processing information. Each neuron is a worker that takes some input, applies its own judgment (weights and bias), and passes a result forward. Width means hiring more workers in the same department — each worker looks at the same set of facts but from a different angle. Depth means adding more departments in a chain — the first department extracts basic patterns, the next combines those into more complex patterns, and so on. A wide network can think about many things at once but has a shallow understanding. A deep network thinks in layers: it builds a hierarchy of ideas, from simple edges to complex objects. But deeper chains are harder to manage — if one worker makes a mistake, it propagates down the line, and errors can grow or vanish. Width is easier to manage but hits a ceiling: no matter how many workers you add, they are all looking at the same level of detail. Depth is more powerful but fragile: you need careful design (like residual connections and normalization) to keep the chain stable.


## An analogy

Imagine you are trying to identify a fruit from its features: color, size, and texture. A wide network is like a panel of experts, each specializing in one feature. One expert looks only at color, another only at size, another only at texture. Each gives a vote, and you combine their votes to decide. If the fruit is a red, round, smooth apple, the panel works well. But if the fruit is a tomato, which is also red and round, the panel might get confused — it has no way to combine its experts' opinions into a higher-level concept like 'this is a fruit, not a vegetable'. A deep network is like a team of detectives: the first group checks color, the second group checks shape and combines color with shape, the third group checks texture and combines that with previous findings, and so on. The detectives can build a complex rule: 'red AND round AND smooth AND small → apple, but red AND round AND smooth AND large → tomato'. Depth lets them combine simple clues into complex conclusions. But detectives in a long chain are hard to coordinate: if the first group makes a small error, it grows as it passes down the line. The panel of experts is easier to manage — each expert works independently — but they cannot see the whole picture. That is the tradeoff: width gives you parallel experts, depth gives you a chain of reasoning. Where the analogy breaks down: in a real team, experts and detectives can communicate freely. In a neural network, information only flows one way, and the chain is fixed. Also, adding more depth does not always help — beyond a certain point, the chain becomes too long to train effectively without special tricks. So the analogy captures the tradeoff but not the mathematical limits of what each can represent.


## Definition

Width vs depth is the design choice of whether to increase the number of neurons per layer (width) or increase the number of layers (depth) in a neural network, trading off representational efficiency against optimisation difficulty.


## How it connects

This concept builds on the artificial neuron (inputs, weights, bias, activation) — you must know what a single neuron does before you can arrange many of them. It also connects directly to **Activation Functions**: without nonlinear activations, a deep stack of layers collapses into a single linear transformation, so depth only works because of the nonlinearity in each neuron. It leads to **Convolutional Networks** and **Transformers**, which are design patterns that specifically manage the balance of width and depth. Understanding width vs depth is the first step to seeing why modern networks are not just wide or just deep but a careful combination of both.


## The mathematics

The mathematics here is about counting: how many computations happen, and how many parameters the network has. We are not doing calculus or algebra; we are doing counting and a little bit of arithmetic. The branch is arithmetic and simple algebra. The concept is: a network has a width (number of neurons per layer) and a depth (number of layers). The total number of parameters — weights and biases — is roughly the width multiplied by depth, plus connections between layers. We also care about how many operations happen when we run the network (forward pass) and how many when we train it (backward pass). More width means more parameters per layer; more depth means more layers, each with parameters. The tradeoff appears when we consider representational power: depth can represent some functions with exponentially fewer parameters than width, but deeper networks are much harder to optimise because the gradient (the direction of improvement) becomes unstable as it travels through many layers.

Let L be the number of layers (depth), and let n_l be the number of neurons in layer l (width per layer). For a fully connected network, the total number of parameters is:

P = Σ_{l=1}^{L} (n_{l-1} × n_l + n_l)

where n_0 is the input dimension. The first term counts weights (each connection between previous and current layer), and the second counts biases. For a fixed total number of parameters, you can choose to spend them on width (increasing n_l) or depth (increasing L). The mathematical insight is that depth can represent some functions with exponentially fewer parameters than width alone.


## A worked example

```
Take a simple network with 2 input neurons, 1 hidden layer, and 1 output neuron. Let the hidden layer have 3 neurons (width = 3) and depth = 1 (one hidden layer). Each connection between input and hidden has a weight, each hidden neuron has a bias, and the output has a weight and bias. Count the parameters.

Step 1: Input to hidden: 2 inputs × 3 hidden neurons = 6 weights   # Each of the 3 hidden neurons receives 2 inputs, so that is 2×3 = 6 weights.
Step 2: Hidden to output: 3 hidden neurons × 1 output = 3 weights   # The output neuron receives 3 inputs, so that is 3 weights.
Step 3: Biases: 3 hidden neurons + 1 output = 4 biases   # Each neuron (hidden and output) has a bias.
Step 4: Total parameters = 6 + 3 + 4 = 13   # Sum of all weights and biases.

Result: The network has 13 parameters. Now, if we increase width to 4 neurons in the hidden layer, the count becomes: input to hidden 2×4=8, hidden to output 4×1=4, biases 4+1=5, total 17. If we instead add a second hidden layer with 3 neurons (depth 2), the count becomes: input to hidden 2×3=6, hidden to hidden 3×3=9, hidden to output 3×1=3, biases 3+3+1=7, total 25. Adding a layer adds more parameters than adding a neuron to an existing layer.
```


## From scratch, in Python

```python
# We'll build two tiny networks: one wide (more neurons per layer) and one deep (more layers).
# Both have the same number of parameters (roughly) to compare how they behave.

# First, define a simple artificial neuron as a function.
# It takes a list of inputs, a list of weights, and a bias.
# It computes the weighted sum (like we did by hand) and returns it.

def neuron(inputs, weights, bias):
    # 'inputs' is a list, e.g. [1, 2]
    # 'weights' is a list, same length, e.g. [0.5, -0.3]
    # 'bias' is a number, e.g. 0.1
    # We'll loop over pairs (input, weight) and add their product.
    total = bias
    for inp, w in zip(inputs, weights):
        total = total + inp * w
    return total

# Now build a forward pass for a network.
# We'll represent the network as a list of layers.
# Each layer is a list of neurons (each neuron is a dict with weights and bias).
# The network takes an input list and returns the output of the last layer.

def forward(network, input_vector):
    current = input_vector
    for layer in network:
        new_current = []
        for neuron_dict in layer:
            # neuron_dict has 'weights' and 'bias'
            output = neuron(current, neuron_dict['weights'], neuron_dict['bias'])
            new_current.append(output)
        current = new_current
    return current

# Build a wide network: 1 hidden layer with 10 neurons.
# Input size = 2, output size = 1.
wide_network = []
hidden_layer_wide = []
for i in range(10):
    hidden_layer_wide.append({'weights': [0.1, -0.2], 'bias': 0.0})
wide_network.append(hidden_layer_wide)
output_neuron = {'weights': [0.1]*10, 'bias': 0.0}  # 10 weights, one per hidden neuron
wide_network.append([output_neuron])

# Build a deep network: 3 hidden layers with 3 neurons each.
deep_network = []
for layer_index in range(3):
    layer = []
    for i in range(3):
        # For the first hidden layer, input size is 2; for others, input size is 3.
        if layer_index == 0:
            layer.append({'weights': [0.1, -0.2], 'bias': 0.0})
        else:
            layer.append({'weights': [0.1, -0.2, 0.05], 'bias': 0.0})
    deep_network.append(layer)
output_neuron_deep = {'weights': [0.1]*3, 'bias': 0.0}
deep_network.append([output_neuron_deep])

# Let's run both with the same input [1, 2].
input_vec = [1, 2]
print("Wide network output:", forward(wide_network, input_vec))
print("Deep network output:", forward(deep_network, input_vec))

# The point is not the output, but the number of parameters and the structure.
# Wide network has: 10 neurons in hidden layer, each with 2 weights + 1 bias = 30 params.
# Output neuron has 10 weights + 1 bias = 11 params. Total = 41.
# Deep network: each of the 3 hidden neurons in first layer has 2 weights + 1 bias = 9.
# Second and third layers: each 3 neurons have 3 weights + 1 bias = 12 each.
# Output has 3 weights + 1 bias = 4. Total = 9 + 12 + 12 + 4 = 37.
# Different architectures, similar parameter count — but they represent different functions.
```


## The same thing, with a framework

```python
In PyTorch, you would define these as sequential layers. A wide network would be a single 'nn.Linear(2, 10)' followed by 'nn.Linear(10, 1)'. A deep network would be 'nn.Linear(2, 3)' followed by several 'nn.Linear(3, 3)' and finally 'nn.Linear(3, 1)'. The library does exactly what our 'forward' function does, but with efficient tensors and automatic differentiation for training. The key is that the architecture choice — width vs depth — is just how you chain these layers. PyTorch makes it easy to experiment with different architectures, but understanding the tradeoff comes from the counting we did by hand.
```


## What the code does

We first define a 'neuron' function that does exactly the weighted sum we practised by hand: take a list of inputs, a list of weights of the same length, and a bias. The 'zip' function pairs each input with its weight, and we add up the products. This is the heart of a neuron. Then we define 'forward' which takes a network (a list of layers, each layer a list of neurons) and an input vector. It runs the input through each layer in order, collecting the outputs of all neurons in a layer to form the next layer's input. This is a full forward pass — just repeated neuron calls. Then we build two networks: 'wide_network' has one hidden layer of 10 neurons — that's width. 'deep_network' has three hidden layers of 3 neurons each — that's depth. Both have a similar number of parameters (41 vs 37). We run both with the same input to see they give different outputs — they are different functions despite similar size. The code makes concrete what the mathematics counted: width and depth are different ways to spend your parameter budget.


## Common misconceptions

**Deeper is always better** — A network with more layers will always be more accurate than a shallower one with the same number of parameters.

Without special tricks like residual connections and batch normalisation, deep networks are very hard to train. The gradient (the signal that tells each weight how to change) either shrinks to zero as it travels through many layers (vanishing gradient) or explodes to huge values. Practically, a deep network can be worse than a shallow one if it cannot be optimised properly. In fact: Depth gives the potential for more expressive representations, but that potential is only realised if the network can be trained. Modern architectures add 'shortcut' connections and normalisation to allow very deep networks to train. Depth and width must be chosen together, and both have limits.

**Width and depth are interchangeable** — A wide network with 100 neurons can do the same job as a deep network with 10 layers, as long as I keep the total number of neurons the same.

There are mathematical results (like the ones cited in the learner's library from Barron Spaces) showing that some functions require exponentially more neurons if you only have width, while depth can represent them compactly. In practice, a wide network may not learn the hierarchical patterns that depth enables. In fact: Width and depth are not equivalent. Depth allows composition of features — simple concepts at early layers combine into complex ones later. Width only provides more parallel processing at a fixed level of abstraction. Some problems cannot be solved efficiently without depth.

**Width is purely a capacity knob** — Adding more neurons per layer just makes the network bigger, nothing else changes.

Adding width also changes the optimisation landscape. A wider layer has more parameters, which can lead to overfitting if not regularised. It can also make the network easier or harder to train depending on the activation function (some activations have better properties with wide layers). In fact: Width affects not only the number of parameters but also the network's ability to represent complex functions at each layer. It is not just a size dial; it changes the function space the network can approximate.

**Depth only increases training time** — More layers just makes the network slower to compute, but the quality of predictions stays the same.

Depth changes what the network can represent. A shallow network can only compute simple combinations of inputs; a deep network can compute hierarchical features (e.g., first edges, then shapes, then objects). If you remove depth, you lose that compositional power entirely, regardless of training time. In fact: Depth is not just a performance cost — it is the mechanism for representation. Without depth, a network cannot learn features that are built from other features.

**The optimal width and depth are fixed numbers** — There is a single best width and depth for all problems, like 512 neurons and 12 layers, and I should use that everywhere.

The best architecture depends on the task, the amount of data, and the compute budget. A model for image classification needs different depth than one for tabular fraud detection. There is no universal setting. In fact: Architecture design is empirical. You start from heuristics (e.g., doubling width after each pooling layer in CNNs) and then experiment. The tradeoff between width and depth is a lever you adjust based on your problem's specific needs.


## In short

Width is how many neurons in each layer; depth is how many layers. Depth lets you build hierarchical representations — simple features composed into complex ones — which is exponentially more efficient in some cases. But depth makes training harder because gradients become unstable. Modern networks balance both: they are deep enough to express complex functions, and wide enough to train reliably. Always consider both when designing a network, and remember that adding depth without special tricks (residual connections, normalisation) often backfires.


## Where this is used

In modern AI, width and depth are the two dials every network architect turns. **Large language models (LLMs) like GPT**: Transformers have a specific width (the dimension of the hidden states, e.g. 4096) and depth (the number of transformer blocks, e.g. 32 or 96). These are the two numbers that define the model's total parameter count. Developers choose both: increasing width adds capacity to represent more complex patterns at each position, while increasing depth allows the model to reason hierarchically — first identifying words, then phrases, then higher-level semantics. In practice, LLMs are both wide and deep, but there is a careful balance: too much width with too little depth leads to a model that memorizes but cannot reason; too much depth with too little width leads to a model that is hard to train and may diverge. **Small language models (running on phones or laptops)**: Here the budget is tight. You cannot have both wide and deep because of memory and compute limits. Often you choose to reduce width (smaller hidden size) and keep depth, because depth gives more reasoning per parameter. For example, a model like TinyLlama is a scaled-down GPT with fewer layers and smaller width, but the depth-to-width ratio is kept similar to the large version to preserve capabilities. **Retrieval-Augmented Generation (RAG)**: In RAG, you have an embedding model (which converts text to vectors) and a generator (an LLM). The embedding model is often a relatively shallow network (e.g., a few layers) because its job is to map text to a fixed-dimensional space, not to reason. Width matters more for the embedding quality — a wider embedding layer captures more nuances. The generator, on the other hand, is deep because reasoning about the retrieved documents requires hierarchy. So width and depth are chosen differently for different parts of the pipeline. **Agentic systems (planning, tool calling)**: Agents are built on LLMs, so the same width-depth tradeoff applies to the underlying model. However, agents also have a 'loop' structure where the model is called repeatedly; the depth of the model determines how well it can plan multiple steps. A deeper model can hold a longer chain of reasoning, which is crucial for tool selection and error recovery. Width, on the other hand, helps the model keep track of many simultaneous goals. Developers often choose a deeper model for agent tasks even if it is slower, because the reasoning depth is more valuable than parallel capacity. **Everyday applied ML (recommendation, vision, forecasting, fraud detection)**: In recommendation systems, you often see wide networks for capturing simple patterns (e.g., which items are frequently bought together) and deep networks for capturing user preferences over time. The 'Wide & Deep' architecture is literally named after this tradeoff — it has a wide linear part and a deep neural part. In vision, convolutional networks are deep (many layers) because they build hierarchical features from edges to shapes to objects; width (number of filters per layer) is also increased but the depth is critical. In forecasting, a shallow network might not capture long-term trends, so depth is used to model temporal hierarchies. In fraud detection, you often need both: width to handle many features at once, and depth to detect complex fraud patterns that span multiple transactions. **What a developer actually writes**: In PyTorch, you write 'nn.Sequential' with a series of 'nn.Linear' layers (or convolutional layers), choosing the numbers. The choice is driven by experimentation and heuristics, not a fixed formula. The breakage when done badly: too wide a network with too little depth may overfit (memorize the training data) because it has too many parameters but not enough abstraction; too deep a network without residual connections or normalization fails to train — the gradients vanish and the network learns nothing. These are the two failure modes you avoid by understanding width vs depth.


---

Read this on the web: https://basavapurushottam.com/learn-ai/width-vs-depth
