# Gradient Descent

Iteratively stepping parameters against the gradient of the loss.

**Rests on:** Computational Graphs

**Leads to:** Diffusion Image Generation, Gradient Obfuscation, GRPO, Hyperparameter Tuning, Inverse Problems, Knowledge Distillation, Lagrange Multipliers, Line Search, Neural Tangent Kernel, Newton Method, Nonsmooth Optimization, Optimal Control, Proximal Gradient Methods, Stochastic Gradient Descent, Training Dynamics

*Mathematical Foundations · Optimization*


## Why learn this

Imagine trying to cook the perfect meal but having no way to tell if your dish tastes good or terrible until it arrives on the table, and even then, only a vague sense of 'meh'. Without a clear signal to guide you, you would just throw ingredients at the wall and hope for the best. Gradient descent is that signal. It is the method used to automatically adjust the settings inside computer programs — from the app that recommends your next video to the system detecting fraudulent credit card charges — so they get better over time. You need to understand this because it explains how machines 'learn'. If you do not grasp how the step size (how big a correction is made) affects the process, you will never be able to fix a program that refuses to improve or learns too slowly.


## The intuition

Picture yourself standing on a vast, foggy hillside at night. You want to find the lowest point in the valley below, perhaps where there is fresh water or shelter. But the fog is so thick you cannot see the bottom. You can only feel the ground directly under your feet. 

You determine which way is 'down' by noticing that the ground slopes downward if you take a step to the left, but rises if you step to the right. So, you walk left. You stop again. The fog still hides the valley floor, but now the ground feels flatter. You realize you are getting closer. You continue walking in the direction where the ground drops away most steeply. Each time you stop, you assess the immediate slope under your boots and take another step downward. This process of repeatedly sensing the local slope and stepping downhill is gradient descent.

The critical variable here is the size of your stride, known as the learning rate. If you take huge strides, hoping to cover ground quickly, you might leap over a small dip in the terrain and land on the other side, further from the bottom than before. You would then have to turn around and climb back down. Conversely, if you take microscopic shuffles, you will eventually reach the bottom, but it might take thousands of years. The 'gradient' is simply the steepness of the slope at your current location, pointing you in the direction of the steepest fall.

In computing, instead of a physical hill, we have a mathematical landscape called the loss surface. The 'height' of this landscape represents an error score (the loss) — how wrong a prediction is. The 'position' you walk across is defined by the weights, which are the adjustable numbers inside the model that determine its behavior. The goal of gradient descent is to navigate this mathematical surface to find the coordinates where the error score is lowest.


## An analogy

The hiker in the fog is the classic analogy for this concept because it perfectly captures the problem of navigating a complex landscape with limited information. A hiker wants to reach the valley floor but cannot see the entire map. Instead, they rely on local cues: the slope under their boots. By always stepping in the direction that feels steepest downward, they make progress without needing to see the destination. The fog represents the complexity of the data; the step size represents the learning rate; and the dip in the ground represents a 'local minimum', a spot that looks like the bottom from your current vantage point but is actually higher than the true valley floor further away. While it seems logical to get stuck in such dips, in complex landscapes with billions of variables, these small dips are often low enough to be acceptable solutions, making the analogy not just illustrative but practically accurate.

This analogy holds up well because it emphasizes that progress is made through small, local adjustments rather than global insight. However, it breaks down slightly when we consider that the 'hill' in computing is not a physical object with friction, but a purely mathematical function where the 'ground' can change shape entirely if the data changes, whereas the hiker's hill remains static.


## Definition

Gradient descent is a method for finding the lowest point of a function by repeatedly moving in the direction of the steepest downward slope, adjusting variables slightly at each step to minimize error.


## How it connects

This concept sits at the foundation of Stochastic Gradient Descent, which is the same walking-down-the-hill technique but applied to random small batches of data rather than all data at once, making the process much faster for large datasets. It also connects directly to Momentum, a variation that adds a 'slingshot' effect by remembering previous steps, helping the model roll through flat areas and small dips without getting stuck too easily.


## The mathematics

Gradient descent is a method for finding the lowest point of a curve or surface. It works by looking at the slope at your current position and taking a small step downhill.

Imagine you are standing on a hillside. You want to get to the bottom of the valley. You cannot see the bottom, so you feel the ground under your feet to determine which way is steepest downward. That direction is given by the gradient. The gradient is a list of numbers that tells you how steep the slope is in every possible direction. If you move in the direction opposite to the gradient (downhill), your height (the loss) will decrease.

The process repeats in steps:
1. Measure the slope at your current position.
2. Take a step in the downhill direction.
3. Repeat until you stop moving significantly.

The size of each step is controlled by a number called the learning rate, written as eta (η). If the learning rate is too small, you take tiny steps and it takes forever to reach the bottom. If the learning rate is too large, you might step over the bottom and land on the other side, potentially bouncing back and forth without settling.

The formula for updating one weight is:
new weight = old weight minus (learning rate times slope)

In symbols:
w_new = w_old - η * slope

Here is what each part means:
- w_old: The value of the weight before this step.
- w_new: The new value of the weight after this step.
- η (eta): The learning rate, a small positive number like 0.01 or 0.1. It scales the size of the step.
- slope: The derivative of the loss function with respect to the weight. It tells you how much the loss changes for a tiny change in the weight. A positive slope means the loss goes up as the weight increases; a negative slope means the loss goes down as the weight increases.

Why do we subtract? Because we want to go downhill. If the slope is positive (going uphill if you increase the weight), subtracting a positive amount makes the weight smaller, moving it downhill. If the slope is negative (going downhill if you increase the weight), subtracting a negative amount is the same as adding, which makes the weight larger, continuing the downhill movement.

For models with many weights, we do this for every weight at once. The gradient is just the collection of all these individual slopes.


## A worked example

Let us walk through one step of gradient descent with real numbers.

Suppose we have a simple model with one weight, w. We want to find the value of w that makes the loss L as small as possible. The loss function is defined as:
L(w) = (w - 3)^2

This is a parabola shaped like a U. The lowest point is at w = 3, where the loss is 0. If w is anything other than 3, the loss is positive.

We start with an initial guess for w. Let us choose:
w_old = 0

We also need to choose a learning rate. Let us choose:
η = 0.1

To take a step, we first need to know the slope at our current position (w = 0). The slope is the derivative of L with respect to w. For the function (w - 3)^2, the derivative is:
slope = 2 * (w - 3)

Let us calculate the slope at w = 0:
slope = 2 * (0 - 3)
slope = 2 * (-3)
slope = -6

The slope is -6. This means that if we increase w slightly from 0, the loss will decrease rapidly (because the slope is negative). To go downhill, we need to move in the direction opposite to the gradient. Since the gradient (slope) is negative, the opposite direction is positive. We add to our weight.

Now we apply the update rule:
w_new = w_old - η * slope
w_new = 0 - (0.1 * -6)

First, multiply the learning rate by the slope:
0.1 * -6 = -0.6

Then subtract this result from w_old:
w_new = 0 - (-0.6)
Subtracting a negative number is the same as adding:
w_new = 0 + 0.6
w_new = 0.6

Our new weight is 0.6. This is closer to the target of 3 than our starting point of 0.

Let us take one more step to show how it continues.
Current weight: w_old = 0.6
Learning rate: η = 0.1

Calculate the new slope at w = 0.6:
slope = 2 * (0.6 - 3)
slope = 2 * (-2.4)
slope = -4.8

The slope is still negative, but smaller in magnitude than before (-4.8 instead of -6). This means the hill is getting flatter as we approach the bottom.

Apply the update rule:
w_new = 0.6 - (0.1 * -4.8)
w_new = 0.6 - (-0.48)
w_new = 0.6 + 0.48
w_new = 1.08

Our weight is now 1.08. It has moved from 0 to 0.6, and then to 1.08. Each step gets smaller because the slope gets smaller as we get closer to the minimum at w = 3.

If we continued this process, the weight would get closer and closer to 3, but it would never quite reach it in a finite number of steps because the steps become infinitely small as the slope approaches zero. This is called convergence.


## From scratch, in Python

```python
# Gradient descent on a simple function: L(w) = (w - 3)^2

# We'll do this with plain Python, no libraries.

# First, define the loss function and its derivative.
# We'll use a function (a reusable block of code) that takes a weight and returns the loss.
def loss(w):
    return (w - 3) ** 2   # the ** means 'to the power of'

# The derivative of the loss with respect to w.
def derivative(w):
    return 2 * (w - 3)

# Set the starting weight and the learning rate.
w = 0.0          # our starting point
learning_rate = 0.1   # step size

# We'll do 10 steps. A 'for' loop runs the indented code that many times.
for step in range(10):
    grad = derivative(w)        # compute the gradient at the current w
    w = w - learning_rate * grad  # update w: move opposite to the gradient
    print(f"Step {step}: w = {w:.4f}, loss = {loss(w):.4f}")

# After 10 steps, w should be close to 3.
```


## The same thing, with a framework

```python
# The same thing with NumPy, for when you have lots of numbers.
# NumPy gives you arrays (lists of numbers) and fast math operations.
import numpy as np

w = np.array([0.0])       # a NumPy array with one element
learning_rate = 0.1

for step in range(10):
    grad = 2 * (w - 3)    # derivative of (w-3)^2, but now as an array
    w = w - learning_rate * grad
    print(f"Step {step}: w = {w[0]:.4f}")

# In a real neural network, w would be a huge array of all the weights,
# and the gradient would come from backpropagation (which you will learn later).
# NumPy simply makes the arithmetic faster when the array has millions of entries.
```


## What the code does

The code starts by defining two functions: `loss` and `derivative`. A function is a block of code that you can call multiple times with different inputs. `loss(w)` computes (w − 3)², which is our valley — the lower the loss, the better. `derivative(w)` computes 2 × (w − 3), which is the gradient at the point w. Then we set our starting point, w = 0.0, and the learning rate, 0.1. The `for` loop runs 10 times: each time it computes the gradient at the current w, then updates w by subtracting the learning rate times the gradient. The `print` statement shows the progress. If you run this, you will see w increasing from 0 toward 3, with the loss shrinking each step and the steps getting smaller as the gradient flattens out near the minimum. This is gradient descent in its purest form — the same structure that trains neural networks, just with a far more complicated loss function that depends on millions of weights.


## Common misconceptions

**The learning rate is the speed of training** — A higher learning rate always makes training faster, and a lower one always makes it slower but safer.

The learning rate controls the step size, not the speed in the sense of wall-clock time. A learning rate that is too large causes the weights to overshoot the minimum and oscillate, or even diverge to infinity. A learning rate that is too small makes progress painfully slow, but guarantees you will eventually get somewhere (for convex problems). The right learning rate is a balance. In fact: The learning rate is the step size. There is a sweet spot: big enough to make steady progress, small enough to avoid overshooting. Learning rate schedules (start large, decay over time) are a pragmatic way to get the best of both worlds.

**Gradient descent always finds the global minimum** — Gradient descent will always find the lowest possible loss, no matter where it starts.

Gradient descent only goes downhill from its starting point. If there are multiple valleys (local minima), it will settle in the first one it finds. Whether that is the global minimum depends on the shape of the loss surface and where you start. In deep learning, however, local minima turn out to be almost as good as the global one in practice, so this is less of a worry than it sounds. In fact: Gradient descent finds a local minimum, not necessarily the global one. In deep networks, local minima are often nearly as good, which is why the algorithm is still used everywhere.

**Gradient descent is only for neural networks** — Gradient descent is a deep learning thing; classic machine learning doesn't use it.

Gradient descent is a general optimisation method. It is used to fit linear regression, logistic regression, support vector machines, and many other models. Even when a closed-form solution exists (like in least squares), gradient descent is often preferred for large datasets because it can handle many examples and many parameters without needing to invert a huge matrix. In fact: Gradient descent is a general-purpose optimiser. It is used in classical machine learning (logistic regression, SVMs) and deep learning alike.

**The gradient is a single number** — The gradient of a loss function is one number that tells you how far to move.

A loss function depends on many weights, and the gradient is a vector — a list of numbers, one for each weight. Each number is the derivative of the loss with respect to that particular weight. The gradient as a whole points in the direction of steepest ascent in the high-dimensional space of all weights. In fact: The gradient is a vector: one derivative per weight. It tells you the direction of steepest ascent; you move opposite to it to descend.

**Gradient descent is a magic bullet that always works** — If training fails, it's because the data is bad or the model is wrong. Gradient descent is reliable and never the problem.

Gradient descent can fail for many reasons: poor initialisation, a learning rate that is too high or too low, vanishing or exploding gradients, or a loss surface that is badly shaped (with plateaus or cliffs). The algorithm is not guaranteed to converge to anything useful in nonconvex problems. In fact: Gradient descent is a heuristic. Its success depends on careful choices of the learning rate, initialisation, and (often) the use of improvements like momentum and adaptive learning rates. Debugging training often means adjusting these, not just blaming the data.


## In short

Gradient descent is the fundamental algorithm for training machine learning models. It works by repeatedly taking steps downhill in the direction of the gradient of the loss function, with the step size controlled by the learning rate. The gradient is a vector of derivatives, one per weight. Getting the learning rate right is the single most important hyperparameter decision: too big and the model diverges; too small and it crawls. Gradient descent finds a local minimum, which in deep learning is usually good enough. Momentum and stochastic variants improve on plain gradient descent in practice. The update rule is: w = w - η * dL/dw.


## Where this is used

Gradient descent is not a niche technique — it is the engine that trains virtually every neural network in production. Here is where it shows up, and how.

**Large language models (LLMs).** When you train GPT-style models, the parameters are the weights of a transformer network — billions of them. Training runs gradient descent on a loss function that measures how well the model predicts the next word in a massive corpus of text. The gradient is computed by backpropagation (a clever way to calculate derivatives through millions of operations), and then each weight is nudged by a small step against that gradient. The learning rate here is not a single fixed number: it typically follows a schedule — it starts small, then climbs, then decays. Getting the schedule wrong can waste an entire multi-million-dollar training run. After training, fine-tuning (adapting a model to a new task) is also gradient descent, but with a much smaller learning rate so the pre-trained knowledge is not destroyed.

**Small language models (on phones and laptops).** When you run a model on a device with limited memory, the same training algorithm applies, but the model is smaller — maybe a few hundred million parameters instead of hundreds of billions. The learning rate and batch size (how many examples are processed before each update) need to be adjusted. Gradient descent still runs, often using a variant called Adam (which adds momentum and per-parameter learning rates). The key constraint is memory: each weight needs a gradient, and the gradient often takes as much memory as the weight itself. Developers use tricks like gradient checkpointing to trade memory for computation, but the underlying loop is still gradient descent.

**Retrieval-augmented generation (RAG).** RAG systems combine a retrieval step (searching a database for relevant documents) with a generation step (an LLM writing an answer). The retrieval step often uses embeddings — vectors that represent the meaning of text — and the embedding model is trained with contrastive loss, which again uses gradient descent. The loss pushes embeddings of similar texts closer together and different texts farther apart. When a developer fine-tunes an embedding model for a specific domain (say, legal documents), they run gradient descent on a dataset of relevant/irrelevant pairs.

**Agentic systems.** Agents that plan, call tools, and act in loops are often built on LLMs that have been fine-tuned with gradient descent on trajectories of successful actions. For reinforcement learning (where the agent learns from rewards), a policy gradient method adjusts the LLM's weights to increase the probability of actions that lead to high reward — again, gradient descent on the policy parameters. The update rule is the same, only the loss function comes from the reward rather than from next-word prediction.

**Everyday applied machine learning.** Recommendation systems (Netflix, Amazon) train on user interaction data, minimising a loss such as mean squared error or binary cross-entropy, using gradient descent. Vision models (face recognition, medical imaging) use gradient descent on cross-entropy loss. Fraud detection models train on imbalanced datasets, using weighted losses and gradient descent to adjust the weights of the classifier. In all these cases, a developer reaches for a framework like PyTorch or TensorFlow, defines the model and the loss, and calls an optimizer — `optim.SGD` or `optim.Adam` — which is simply a packaged gradient descent loop. The developer does not write the update rule by hand, but the underlying arithmetic is exactly what we worked through above.

**What actually breaks when done badly?** The most common failure is divergence: if the learning rate is too high, the weights can jump back and forth with ever-increasing magnitude, and the loss goes to infinity. If the learning rate is too low, training takes forever. A classic practice is the 'gradient check' — verifying that a numeric approximation of the gradient (computed by slightly nudging the weights and measuring the loss change) matches the analytic gradient from backpropagation. The learner's library records this from *Deep Learning From Scratch with Python*: 'Matching the numeric gradient is the standard sanity check for any backprop implementation — write it before you trust the network.' In production, a missing gradient check can mean silently training a model that never learns — which is far worse than a crash, because it looks like it might be working.


---

Read this on the web: https://basavapurushottam.com/learn-ai/gradient-descent
