# Stochastic Gradient Descent

Estimating the gradient from a minibatch instead of the full dataset, trading exactness for far more updates per unit of computation.

**Rests on:** Gradient Descent

**Leads to:** Double Descent, SISA Sharded Training, Stock Price Prediction

*Mathematical Foundations · Optimization*


## Why learn this

This method is the engine behind almost every modern AI system you use, from voice assistants on your phone to large language models that write text. Without it, these systems could not learn from data. Understanding this process lets you see how a computer goes from knowing nothing to making predictions, and gives you the context needed to understand related ideas like 'momentum' (which speeds up learning) or 'Adam' (a more advanced version of this method used in practice). It is also essential for reading training graphs and understanding what happens when we adjust an existing model.


## The intuition

Imagine you are hiking down a steep mountain in thick fog. You cannot see the peak or the path ahead; you can only feel the slope of the ground right under your feet. To get down, you take a step in the direction that feels steepest downward, then stop to feel the ground again and take another step. You will not find the perfect path — you might step over a small ridge into a different valley — but you will reach the bottom far faster than if you stopped to map the entire mountain before every single move.


## An analogy

Think of a student trying to master a subject for an exam using a textbook with thousands of practice problems. A 'perfect' approach would require solving every single problem perfectly, analyzing all results at once, and then adjusting their study plan. This is thorough but incredibly slow. Instead, imagine the SGD approach: The student grabs ten random problems, solves them, sees what they got wrong, and adjusts their understanding for that evening. Then they pick ten different random problems and repeat. Their daily study plan is noisy — those ten problems might not represent the whole subject perfectly — but because they learn every day rather than once at the end of the month, they make rapid progress over weeks.


## Definition

Stochastic Gradient Descent is a method for improving predictions by estimating the direction to adjust based on a small, random sample of data, then updating the model's settings in that direction, and repeating this process many times to gradually reduce error.


## How it connects

This concept belongs to **Optimization**, which is the general study of finding the best possible outcome from many choices. SGD is a specific technique within this field that uses randomness to speed up the search for better settings. It connects closely to **Learning Rate**, which determines how big each step is during this process, and **Loss Function**, which is the measure of error we are trying to minimize.


## The mathematics

This concept rests on **calculus**, specifically the derivative. Calculus is the mathematics of change: it answers the question 'if I nudge this input just a tiny bit, how much does the output change?' Without calculus, we would have no idea which direction to turn the dials of a model. The derivative is the exact slope of a curve at a point — like measuring the steepness of the hill right where you are standing.

But we do not use the derivative on a simple curve; we use it on a function that depends on millions of dials (the model's parameters) and millions of data examples. The full gradient is a giant list of slopes, one for each dial, and computing it exactly requires looking at every single training example. That is expensive.

The key insight of SGD is to replace the exact gradient with an estimate, computed from a random minibatch. The estimate is noisy — it jumps around — but it is unbiased: on average, it points in the same direction as the true gradient. So we take a step using the noisy estimate, throw it away, and compute a new noisy estimate from a different minibatch. Over many steps, the noise cancels out, and the average direction of travel is correct.

The cost of this noise is that the path to the bottom is wobbly and may not follow the steepest route. The benefit is enormous: many more steps in the same amount of time. In practice, this speed means models that would take centuries to train become trainable in days.

w_new = w_old − η × (1/m) × Σ_{i=1..m} ∇_w L(x_i, y_i, w_old)

Where:
- w_old and w_new: the model's parameters before and after this step.
- η (eta, pronounced 'ay-ta'): the learning rate, the size of the step.
- m: the minibatch size (e.g., 32).
- Σ is a sum symbol, meaning 'add up everything after it'.
- ∇_w (nabla, pronounced 'del-w'): the gradient with respect to w — a list of slopes, one for each parameter.
- L(x_i, y_i, w): the loss (error) of the model on the i-th example, given its input x_i and true answer y_i.
- The whole (1/m) × Σ is the average of the gradients over the minibatch — the estimated gradient.


## A worked example

```
We start with a single parameter, call it w. The loss (error) as a function of w is just a curve: for this example, it is a bowl-shaped curve with its minimum at w = 2. So the true optimal value is w = 2, and we want SGD to find it. Our learning rate (the size of each step) will be 0.1. Our dataset has 100 examples, and we will use a minibatch of size 10. The true gradient of the loss at any w is 2 × (w − 2). At the start, w = 0.

Step 1: True gradient at w=0 is 2 × (0 − 2) = −4, so the exact gradient points right (positive direction).   # The derivative tells us the slope: −4 means that increasing w by 1 would decrease the loss by 4.
Step 2: SGD picks 10 random examples. The estimated gradient might be −3.8, −4.2, or −3.5 (noisy). Here, it estimates −3.8.   # Instead of the true −4, we use an estimate. This is the 'stochastic' part.
Step 3: SGD update: w_new = w − (learning_rate × estimated_gradient) = 0 − (0.1 × (−3.8)) = 0 + 0.38 = 0.38.   # A learning rate of 0.1 means we move only 10% of the way in the direction suggested.
Step 4: With a different minibatch, the next estimate might be −3.6. Then w = 0.38 − (0.1 × (−3.6)) = 0.38 + 0.36 = 0.74.   # Each step uses a fresh random minibatch, so the estimate changes.

Result: After many steps, w will approach 2, but never settle exactly. It will wiggle around 2 (e.g., 1.96, 2.03, 1.99), because each estimate is noisy. The wiggling is harmless — the solution is found within a small tolerance.
```


## From scratch, in Python

```python
# No frameworks, just plain Python.
# We simulate a dataset and a simple loss.

def loss(w):
    """Loss as a function of w, a quadratic curve."""
    return (w - 2) ** 2

def gradient(w):
    """True gradient: 2*(w-2). We'll use this only to generate noisy estimates."""
    return 2 * (w - 2)

# Simulate a dataset of 100 examples, each with a 'true' gradient that has noise.
import random

true_gradient_at_0 = gradient(0)
dataset = [true_gradient_at_0 + random.gauss(0, 0.3) for _ in range(100)]

# Hyperparameters
learning_rate = 0.1
batch_size = 10
epochs = 20  # how many times we go through the whole dataset

w = 0.0

for epoch in range(epochs):
    # Shuffle the dataset so each minibatch is random
    random.shuffle(dataset)
    for start in range(0, len(dataset), batch_size):
        # Get one minibatch
        batch = dataset[start:start+batch_size]
        # Estimate the gradient as the average of the batch
        estimated_gradient = sum(batch) / len(batch)
        # Take a step
        w = w - learning_rate * estimated_gradient

print("Final w:", w)  # Should be close to 2.0
```


## The same thing, with a framework

In practice, you would use a library like PyTorch. The steps are the same, but the library handles the math:
```python
import torch

# Initialize a parameter
w = torch.tensor([0.0], requires_grad=True)

# Optimizer (SGD)
optimizer = torch.optim.SGD([w], lr=0.1)

# Data loader would feed minibatches
for each_minibatch:
    loss = (w - 2) ** 2       # Your loss function
    optimizer.zero_grad()     # Clear old gradients
    loss.backward()           # Compute gradients (the derivative)
    optimizer.step()          # Update w: w = w - lr * gradient
```


## What the code does

The from-scratch code is a faithful miniature of real training. Let's go through it:

- We define a loss function `loss(w)` that measures how wrong a parameter value is. A perfect value (2.0) gives zero loss.
- `gradient(w)` is the derivative — the slope of the loss at w. We are cheating by using it to generate a dataset, but in real life you do not know it; you only have noisy estimates.
- The `dataset` is a list of 100 noisy gradient estimates. In a real setting, each 'estimate' would come from evaluating the model on one data point, but here we simulate it with random noise via `random.gauss`.
- The outer loop (`epoch`) runs 20 times. Each pass over all 100 examples is one epoch.
- The inner loop grabs a minibatch of size `batch_size` (10) using list slicing. These are the examples we will use for one step.
- `estimated_gradient` is the average of the 10 noisy gradients. This is the key step: we are approximating the true gradient with a small sample.
- The update line `w = w - learning_rate * estimated_gradient` is exactly the SGD rule. We step opposite the gradient to reduce loss.
- After many steps, w should hover near 2.0.

The framework code (PyTorch) does the same thing: it computes the derivative automatically (`backward()`) and updates the parameter (`step()`). The wheel is the same — we just do not have to build it ourselves.


## Common misconceptions

**SGD is an approximation and therefore worse than exact gradient descent** — 'If the gradient is noisy, the result must be worse. Why not compute the exact gradient every time?'

This misses the central trade-off. Exact gradient descent takes one step per full pass over the data. SGD takes (dataset size / batch size) steps in the same time. Often, the number of steps matters far more than their exactness. As the learner's notes record, 'gradient noise is not purely a cost — it also helps escape sharp minima'.

The noise can actually push the parameters out of a sharp, narrow valley that would generalise poorly, into a wider, flatter valley that generalises better. This is an empirical benefit, not just a tolerable nuisance. In fact: SGD is not a degraded version of gradient descent — it is a different algorithm with its own strengths. The extra steps it can take in the same wall-clock time usually outweigh the noise, and the noise itself can help find better, more generalisable solutions.

**The learning rate is unrelated to the batch size** — These are two independent dials; I can set them however I want.

The learning rate controls how much a single gradient estimate moves the parameters. A larger batch gives a less noisy, more reliable estimate — so you can afford a larger step. Conversely, a tiny batch is noisy and requires a small learning rate to avoid bouncing too far. As the learner's notes say, 'batch size interacts with learning rate; scaling one usually requires scaling the other.' In fact: In practice, the two are tightly coupled. A common rule is to scale the learning rate in proportion to the batch size (or its square root). If you double the batch size and leave the learning rate fixed, each step will not be twice as effective — and your total training time may not improve.

**A bigger batch size is always better** — I have enough memory on my GPU, so why not use a huge batch? It gives more accurate gradients.

Beyond a point, larger batches stop helping in terms of compute efficiency. The estimate becomes accurate, but you are putting so many examples into each step that you take very few steps in total. Also, very large batches have been observed to converge to sharp minima that generalise worse, partly because they lose the regularising effect of noise. In fact: Batch size is a hyperparameter to tune, not a knob to set to maximum. A moderate batch (e.g., 32–256) is often a good default. The right size balances compute efficiency, memory, and the noise that can aid generalisation.

**Adam and SGD are completely different algorithms** — Adam is a magical algorithm that has nothing to do with plain SGD.

Adam builds directly on SGD. It takes the same gradient step, but it adapts the learning rate per parameter based on the history of past gradients. It also uses momentum-like averaging. The core idea — estimate the gradient from a minibatch and step against it — is identical. In fact: Adam is an enhancement of SGD, not a replacement from a different world. Many practitioners even treat Adam as 'SGD with per-parameter adaptive learning rates + momentum.' The same intuitions about batch size and learning rate still apply.

**The noise in SGD is always a bad thing** — Noise only causes wasted steps and unstable training.

The learner's own notes record the opposite: 'Gradient noise is not purely a cost — it helps escape sharp minima.' Sharp minima tend to generalise poorly. The noise lets the parameters wander out of them into wider valleys where the loss is flat, and where test performance is better. In fact: The noise is a double-edged sword. It can slow convergence and cause wobble, but it can also act as a regulariser and help the model land in better generalising regions.


## In short

Stochastic gradient descent (SGD) trains models by taking many small, noisy steps. Instead of computing the exact gradient from all data (expensive), it estimates the gradient from a small random minibatch (cheap). The learning rate sets the step size; the batch size sets the noise level. The noise can even be beneficial — it helps the model escape sharp minima. This simple loop — estimate, step, repeat — is the engine of virtually every neural network training run, from a phone's wake-word detector to a 70-billion-parameter LLM.


## Where this is used

**Large language models (LLMs).** Training a model like GPT takes months on thousands of GPUs. The data is terabytes. If we tried to compute the exact gradient, each step would require scanning every document — impossibly slow. Instead, the training is a long series of SGD steps on minibatches of a few thousand sentences at a time. The learning rate and batch size are tuned carefully; a learning rate too large makes the training diverge, too small and it crawls. Even the 'fine-tuning' of an existing model (e.g., teaching it to follow instructions) uses SGD on a small dataset. A developer writes a training loop with PyTorch, exactly like the one above, but with millions of parameters — the library's `loss.backward()` and `optimizer.step()` are the same calls.

**Small models on devices (TinyML).** A model that runs on a smartwatch or a hearing aid cannot be huge. Its training, though, often happens on a server, using SGD with small minibatches. The same principle applies: the device's tiny model is trained offline, and then the trained weights are loaded onto the device. For on-device learning (e.g., a phone keyboard learning your typing style), SGD updates happen on the device itself with a few recent keystrokes as the minibatch. The batch size is tiny (maybe 1), which is a special case of SGD called 'online learning' — update after every single example.

**Retrieval-augmented generation (RAG).** RAG systems retrieve the most relevant documents from a database and feed them to an LLM to answer a question. The retrieval is often done by comparing embeddings — lists of numbers that capture meaning. The embedding model is trained with SGD, often with a loss that pushes similar texts closer together. The developer using RAG does not usually train the embedding model from scratch, but they might fine-tune it on their own document set using SGD, so that the embeddings better match their domain.

**Agentic systems.** Systems that plan and call tools (e.g., an agent that books a flight) are usually built by combining a pre-trained LLM with a set of tools. The LLM is trained with SGD; the agent's 'planning' is just the LLM's next-token generation, guided by prompts and possibly fine-tuned with SGD on a dataset of good plan-tool sequences. So while the agentic system's logic is not trained, every neural module inside it (the LLM, the vision encoder) is a direct product of SGD.

**Everyday applied ML (recommendation, vision, fraud detection, speech).** A fraud detection model is trained on millions of transactions. Each minibatch has a few hundred transactions. SGD updates the model's weights to recognise fraud patterns while slowly reducing the loss. In vision (e.g., detecting defects on a factory line), a pre-trained convolutional network is fine-tuned with SGD on a small set of labelled images. In speech, wake-word detection (the 'Hey Siri' model) is trained with SGD on a dataset of audio clips.

**What breaks when it is done badly.** If the learning rate is too high, the loss explodes to infinity in the first few steps — the training diverges. If too low, the model barely learns, wasting compute. If the batch size is too large, each step is slow and the model might overfit to the training data; if too small, the gradient noise makes training unstable. A common recipe is to scale the learning rate with the batch size: if you double the batch size, double the learning rate, to keep the same effective step size. Badly chosen hyperparameters are the most common cause of a model that 'won't learn' — it is not the architecture that is wrong, it is the optimisation.


---

Read this on the web: https://basavapurushottam.com/learn-ai/stochastic-gradient-descent
