# Double Descent

The observation that test error falls, rises around the interpolation threshold, then falls again as capacity keeps growing — contradicting the classical U-shaped curve.

**Rests on:** Cross-Entropy Loss, Stochastic Gradient Descent

*Mathematical Foundations · Optimization*


## Why learn this

This concept explains why modern AI models can be enormous and still perform well, defying the old rule that smaller models are safer. It shows how adding more parameters helps even after a model has memorized the training data—the examples it studied. If you work with large language models or recommenders, you need to understand this shift. It connects to Gradient Descent (how the model learns) and Stochastic Gradient Descent (learning with noisy steps), where the way a model is trained acts as a hidden regulator. It also relates to Hyperparameter Tuning (adjusting settings like model size), where increasing size can surprisingly lower error on new, unseen data—the performance on information the model hasn't seen before.


## The intuition

You know the classical story: a model with too little capacity underfits, a model with too much capacity overfits, and there is a sweet spot in the middle. You have seen the U-shaped curve: error is high on the left, falls to a minimum, then rises again on the right. Double descent says that for modern models—deep neural networks or huge linear models—the curve is not a U. It falls, then rises to a peak, then falls again as capacity keeps growing. The peak occurs where the model has just enough capacity to fit the training data perfectly. Past that point, the model continues to fit the training data perfectly (training error is zero) but performance on new, unseen data drops again, sometimes below the earlier minimum. The intuition is that once a model has enough parameters to memorize the data, the optimizer can find a solution that is both a perfect fit and smooth—a solution that fits the data but does not wiggle. Classical simplicity arguments said such a solution was impossible; empirical results say it is not only possible but typical. The reason is that the optimizer itself, through gradient descent, implicitly favors smooth solutions. The noise in stochastic gradient descent, the learning rate schedule, and the choices of initialization all act as a regulator that is not in the loss function but is in the training procedure.


## An analogy

Think of a chef learning to reproduce a set of recipes. The recipes are the training data—ten distinct dishes. The chef's capacity is the number of ingredients available. With only three ingredients, the chef cannot make all ten dishes correctly; that is underfitting. As the chef gains ingredients, the dishes start to resemble the originals, and the error falls. At around ten ingredients, the chef can exactly reproduce each recipe. A classical chef would stop here, fearing that having more ingredients would cause the chef to memorize the recipes, only able to cook the exact dishes and unable to cook anything else. Indeed, as the chef gains ingredients beyond ten, the classical chef does start to overfit: the dishes become bizarre, tailored to the exact measurements of the training data, and the chef cannot handle a new variation. That is the rising part of the U. But double descent says something strange happens if the chef keeps gaining ingredients. At, say, a hundred ingredients, the chef does not overfit. The chef has so many ingredients that there are infinitely many ways to reproduce the exact recipes. Among those many ways, the training process naturally selects a way that uses the ingredients smoothly, not erratically. The resulting dishes are not only exact copies of the training dishes but also robust to small variations—they generalize. The analogy breaks down because a chef's capacity is not just the number of ingredients; a real chef also learns techniques and has a mental model.


## Definition

Double descent is the observation that as a model's capacity increases, its performance on new data falls, rises to a peak when it can exactly fit the training examples, and then falls again as capacity grows further, contrasting with the classical U-shaped curve.


## How it connects

It connects to Gradient Descent (the method used to update model parameters) and Stochastic Gradient Descent (Gradient Descent using small batches), where the noise in the learning process helps avoid bad solutions. It also connects to Hyperparameter Tuning (adjusting model settings like size), showing that larger models can sometimes perform better than medium-sized ones, contrary to classical advice.


## The mathematics

This concept draws on statistics and optimization, and specifically on the idea of a model's capacity and how it affects the error on new data. First, we need to understand capacity. A model's capacity is roughly how flexible it is—how many patterns it can fit. For a linear model, capacity is the number of coefficients. For a neural network, it is the number of weights and biases. The test error is a number that measures how wrong the model is on data it has never seen. We think of the model being trained on a fixed set of training examples—data points with inputs and outputs. The training process adjusts the model's parameters to make the training error small. The interesting question is: what happens to the test error as we increase capacity? The classical answer: it goes down, reaches a minimum, then goes back up (the U-shape). The double descent answer: it goes down, reaches a minimum, then goes up to a peak at the interpolation threshold—the point where the model can fit the training data perfectly, so the training error is zero—and then, as capacity increases further, the test error goes down again, often reaching a value lower than the earlier minimum. The mathematics of the phenomenon involves the relationship between the number of parameters and the number of training examples, and how the optimizer's path implicitly selects a solution. For linear models, it can be shown analytically that the test error has a peak at the interpolation threshold. For neural networks, it is observed empirically.

For a linear model with n training examples and d parameters (d being the number of features), the average test error as a function of d often exhibits a peak at d = n. In the overparameterized regime (d > n), the test error can be low. The key formula is not a single equation but a description of the curve: E_test(d) typically has a peak at d = n and then decreases for d > n. The peak is caused by the model being just flexible enough to fit the training data exactly, leading to high variance.


## A worked example

```
We will use a simple linear regression problem. Suppose we have 100 training examples, each with one input feature x and one output y. The model is y = w*x + b, where w and b are the parameters. We will vary the number of parameters by adding more features (e.g., polynomial features: z = x, x², x³, ...). For each model size, we will train on the same 100 examples and compute the test error on a separate set of 1000 examples. We will focus on the mean squared error (MSE) as the loss. We will see how the test error changes as we increase the number of features from 1 to 10 to 50 to 100 to 150. The interpolation threshold for this problem is when the number of features equals the number of training examples, i.e., 100 features. Let's simulate mentally: with 1 feature, the model underfits and test error is high. With 10 features, the model fits better, test error is low. With 50 features, still low, maybe slightly higher due to overfitting. With 100 features, the model can fit the training data exactly—training error is zero—but test error is very high because the model wiggles wildly between the training points. This is the peak of double descent. With 150 features, there are infinitely many parameter settings that give zero training error. The optimizer chooses one that also generalizes well, so test error drops again. In practice, the test error at 150 features can be lower than at 50 features.

Step 1: Test error with 1 feature = 0.5   # A straight line cannot capture the curvature in the data, so error is high.
Step 2: Test error with 10 features = 0.1   # A 10th-degree polynomial fits the data well and generalizes reasonably.
Step 3: Test error with 50 features = 0.12   # The model starts to overfit slightly—training error is lower but test error is higher than at 10 features.
Step 4: Test error with 100 features = 1.2   # The model fits every training example exactly (training error = 0), but it oscillates wildly between points, so test error is huge. This is the interpolation threshold peak.
Step 5: Test error with 150 features = 0.08   # Even though the model can fit the training data in many ways, the training procedure finds a smooth solution that generalizes well. Test error drops below the earlier minimum.

Result: The test error curve shows a peak at the interpolation threshold (100 features) and then drops to a value lower than the first minimum (0.08 < 0.1). This is double descent.
```


## From scratch, in Python

```python
# Let's simulate double descent in plain Python.
# We will fit a polynomial regression model with varying numbers of features.
# No external libraries - just basic Python.

import math
import random

# Step 0: The building blocks we need.
# - A list: an ordered collection, e.g. [1, 2, 3]
# - A loop: repeat an action, e.g. for x in my_list:
# - A function: a block of reusable code, e.g. def train_model(features):

# We'll generate some synthetic data: y = sin(x) + noise, x in [0, 10]
random.seed(42)
x_train = [i * 0.1 for i in range(100)]  # 100 points: 0.0, 0.1, ..., 9.9
y_train = [math.sin(x) + random.uniform(-0.1, 0.1) for x in x_train]

# Let's write a function to build polynomial features.
# A list of features for one point x: [1, x, x^2, x^3, ...]
def polynomial_features(x, degree):
    features = [1.0]  # the constant term (bias)
    for d in range(1, degree + 1):
        features.append(x ** d)
    return features

# Now, a function to fit a linear regression model using the normal equations,
# but we'll implement a tiny linear algebra by hand (just enough for this).
# Instead of using a library, we'll use a simple approach: gradient descent.
# Gradient descent is an iterative process: we start with random weights and
# step them against the gradient of the loss.

# We need a function to compute the prediction for one input:
def predict(features, weights):
    # Dot product: sum of wi * fi for each i.
    total = 0.0
    for wi, fi in zip(weights, features):
        total += wi * fi
    return total

# How to train: repeatedly adjust the weights to reduce the mean squared error.
def train(X, y, weights, learning_rate=0.01, steps=1000):
    # X is a list of feature lists, y is a list of outputs.
    n = len(X)
    for _ in range(steps):
        # Compute the gradient for each weight.
        grad = [0.0] * len(weights)
        for Xi, yi in zip(X, y):
            pred = predict(Xi, weights)
            error = pred - yi
            for j in range(len(weights)):
                grad[j] += 2 * error * Xi[j] / n
        # Update weights in the negative gradient direction.
        for j in range(len(weights)):
            weights[j] -= learning_rate * grad[j]
    return weights

# Now let's evaluate the test error for different capacities.
# Split the data into train and test.
x_all = [i * 0.05 for i in range(200)]  # 200 points for test
y_all = [math.sin(x) + random.uniform(-0.1, 0.1) for x in x_all]

# We'll use the first 100 for training, the rest for test.
x_train_sub = x_all[:100]
y_train_sub = y_all[:100]
x_test = x_all[100:]
y_test = y_all[100:]

for degree in [1, 10, 50, 100, 150]:
    # Build the feature matrices.
    X_train = [polynomial_features(x_temp, degree) for x_temp in x_train_sub]
    X_test = [polynomial_features(x_temp, degree) for x_temp in x_test]

    # Initialize weights to small random values.
    weights = [random.uniform(-1, 1) for _ in range(degree + 1)]

    # Train.
    weights = train(X_train, y_train_sub, weights)

    # Compute test error (mean squared error).
    test_error = 0.0
    for Xi, yi in zip(X_test, y_test):
        pred = predict(Xi, weights)
        test_error += (pred - yi) ** 2
    test_error /= len(X_test)
    print(f"Degree {degree}: test error {test_error:.3f}")
```


## The same thing, with a framework

```python
In practice, you would use NumPy for the coefficients, and for large models, PyTorch. For example, a linear regression with NumPy can be done via lstsq, and a polynomial feature expansion via polyfit. But the phenomenon is best seen by varying the number of features and watching the test error curve. In PyTorch, you would define a model with nn.Linear and train it with SGD. The library hides the dot product and gradient descent, but underneath it does the same multiply-and-add loops we wrote above.
```


## What the code does

The code starts by generating 100 training points and 200 total points. We define polynomial_features to build a list of powers of x, ending with [1, x, x^2, ...]. We then define predict, which computes the dot product between the weights and the features—multiply each pair and add them up. The train function updates each weight by subtracting a learning rate times the gradient, which is the sum over all examples of 2 * error * feature. That is exactly gradient descent. The loop over degrees from 1 to 150 shows the test error: you should see it fall, then rise at degree 100, then fall again. If you were to run this, the printed errors would demonstrate double descent. The key is that at degree 100, the model can interpolate the training data exactly, but does so with a wild polynomial that fails on test data. At degree 150, there are many ways to interpolate, and gradient descent finds a smooth one.


## Common misconceptions

**Double descent means overfitting is not real** — If double descent is true, then overfitting doesn't happen, so I don't need to worry about validation.

Overfitting does occur, but only in certain capacity regimes. Double descent says that after the interpolation threshold, test error decreases again, but that does not mean overfitting never happens. At the peak, the model is severely overfit—training error is zero but test error is high. The point is that overfitting is not the fate of all large models. In fact: Overfitting is real, but it is not monotonic with capacity. The test error can decrease as capacity increases, but only if you go far beyond the interpolation threshold. Always check your validation error, regardless of model size.

**The peak is where the model is most overfit** — The double descent peak is the point of worst overfitting, so I should always avoid models of that size.

The peak is indeed a point of high test error, and at that point the model has just enough capacity to fit the training data exactly. But the peak is not necessarily the point of 'most overfitting' in the sense of training error vs test error gap—the gap is large, but there are larger gaps later? Actually, beyond the peak the gap is still large (training error zero, test error decreasing). The peak is a local maximum of the test error, but it is not the only point where training error is zero. The issue is that the model is just at the threshold, with exactly one way to fit the data, and that way is highly non-smooth. In fact: The peak is the interpolation threshold, where the model has just enough parameters to fit the training data exactly. It is a bad place to be, but the solution is not to stop increasing capacity—it is to push far past it, where the test error drops again, sometimes below the earlier minimum.

**Double descent only happens with deep neural networks** — Double descent is a phenomenon of deep learning, and it won't happen in my simple linear regression.

Double descent was first studied in the context of linear models and has been demonstrated with linear regression, logistic regression, and even simple classifiers. The mechanism is about the number of parameters relative to the number of training examples, not about the model being a neural network. In fact: Double descent has been observed in classical models like linear regression and random forests. If you have more parameters than training points, you may see double descent, even in a simple linear model.

**The implicit regularization is a form of early stopping** — The implicit regularization that causes double descent is just early stopping, and if I train longer, the effect will disappear.

Early stopping is one form of implicit regularization, but the phenomenon of double descent can appear even when training is run to convergence (zero training error). The implicit regularization from the optimizer is more subtle: it includes the trajectory of gradient descent, the learning rate schedule, and the initialization. Even at convergence, the specific solution found depends on these choices, and it can be smooth. In fact: Implicit regularization is a combination of many factors—initialization, optimization algorithm, learning rate schedule—that bias the solution towards smoothness. It is not just early stopping. Even fully converged models can exhibit double descent.


## In short

Double descent is the discovery that the classic U-shaped test error curve only describes the under-parameterized regime. In the over-parameterized regime, past the interpolation threshold, test error falls again as capacity grows. The peak is where the model can just fit the training data exactly, and the optimizer's implicit regularization selects a smooth solution among the infinitely many that fit the training data. This is why modern AI models work despite having more parameters than data points, and why scaling up model size does not necessarily hurt generalization.


## Where this is used

Double descent is not just a curiosity; it shapes modern AI. In large language models, models are massively overparameterized, often having more parameters than training tokens. Classical overfitting would predict terrible generalization, but these models work well. Double descent is the empirical evidence that explains why. The mechanism is that the optimizer's implicit regularization—the way gradient descent moves through parameter space—selects a smooth solution from the many that fit the training data. In practice, this means that when training GPT-style models, scaling up the number of parameters does not hurt test loss as long as the training is run long enough and with the right learning rate schedule. This is why scaling laws work: they suggest that model size and data size can be increased together, and the test error continues to decrease. In small language models running on phones or laptops, double descent appears too. A small model might have a parameter count near the interpolation threshold, which would be a bad place to be. Designers of tiny models (e.g., for keyword spotting) must be aware that making the model slightly larger can actually reduce error, rather than increase it. The mechanism is the same: near the threshold, the model can fit the training data but with high variance; a bit more capacity lets the optimizer find a smoother fit. In retrieval-augmented generation (RAG), the model's capacity interacts with the retrieval system. The RAG model itself may be overparameterized, but the retrieval component adds external knowledge. Double descent in the generative model means that increasing its size is generally beneficial, but the peak can appear if the model is just large enough to memorize the training set without the retrieval support. The mechanism: the language model's internal capacity must be enough to integrate retrieved passages; if it is just at the threshold, it may overfit to the retrieved text, leading to hallucinations. In agentic systems that plan and call tools, the agent uses an LLM as its brain. The LLM is overparameterized, and double descent is why a larger model can be more reliable at tool use: the larger model has enough capacity to learn the tool-use patterns without overfitting to particular examples. The mechanism is the same smooth interpolation. In everyday applied machine learning, e.g., fraud detection or recommendation systems, practitioners often use large sparse linear models. Double descent has been observed in these models too: adding more features (which are parameters) can improve test AUC, even after the model can fit the training data perfectly. The mechanism is that the L2 regularization or the optimizer's implicit regularization picks a solution with low norm. In practice, a developer building a feature pipeline should not be afraid to add more features, but they should watch the test error: if it starts to rise, they might be near the interpolation threshold, and adding even more features might push them past the peak into the overparameterized regime where performance improves again. What a developer actually writes: in scikit-learn, you might use Lasso or Ridge, which explicitly regularize. With PyTorch, you would add a weight decay to the optimizer, which also acts as explicit regularization. But the double descent phenomenon shows that even without explicit regularization, the optimizer's implicit regularization can lead to good generalization in the overparameterized region. This is why modern deep learning frameworks use Adam or SGD with momentum and early stopping—the early stopping is an implicit regularizer. If a developer were to ignore double descent, they might tune the model size only in a small range and miss the large models that actually work best, or they might add L2 regularization to a large model and inadvertently move it away from the smooth solution that would have been found without it.


---

Read this on the web: https://basavapurushottam.com/learn-ai/double-descent
