# Training Dynamics

How loss, gradients and representations evolve over the course of training, including warmup, plateaus, sharp drops and the effect of schedules.

**Rests on:** Computational Graphs, Gradient Descent

**Leads to:** Catastrophic Forgetting

*Mathematical Foundations · Optimization*


## Why learn this

When you train a machine learning model, you are not writing code; you are tuning a process. Understanding training dynamics gives you the ability to look at a graph of error over time and instantly diagnose what is going wrong. Without this knowledge, you are adjusting settings like a blindfolded mechanic turning knobs on an engine. With it, you can see exactly why a model is stuck, why it crashed, or why it stopped improving. This understanding prevents wasted compute time and resources by allowing you to fix problems before they become catastrophic failures.

Consider this concrete situation: You start training a text prediction model. After 100 steps, the error (loss) has not moved at all. It is flat at 5.2. A day later, you check again, and the error is still 5.2. Without knowing what to look for, you might wait another week. With this knowledge, you recognize that the model is trapped in a 'plateau'—a flat area where the signal telling it how to improve has vanished. You can then take specific actions, such as changing how large your updates are, to get the process moving again. This is the difference between guessing and diagnosing.


## The intuition

Imagine you are trying to find the lowest point in a vast, foggy valley while standing blindfolded. You cannot see the bottom. All you can do is stand on the ground, feel which way the slope tilts under your feet, and take a step downhill. 

This process has several key components:

1. The Slope (Gradient): At any moment, you can tilt your body or use a level to find the direction where the ground drops away most steeply. This direction is called the gradient. It tells you which way to go.

2. The Step Size (Learning Rate): How far do you step? If you take a tiny step, you make slow progress but stay safe. If you take a giant leap, you might reach the bottom quickly—or you might overshoot it and land on the other side of the valley, higher up than before.

3. The Altitude (Loss): After each step, you measure your height above the true bottom of the valley. This measurement is the loss. Your goal is to make this number smaller.

4. The Plateau: Sometimes, you reach a wide, flat shelf in the middle of the valley. The ground feels level in every direction. There is no steep slope to guide you. You stand there for many steps, making no progress, because the signal (the slope) has disappeared. This is a common problem called 'vanishing gradients.'

5. The Schedule: Deciding how big your steps should be over time is critical. Early on, you want large steps to cover ground quickly. But as you get closer to the bottom, large steps become dangerous; you will bounce around the bottom without settling. Therefore, most training plans include a 'schedule'—a rule for shrinking your step size over time.

Two common schedules are:
- Cosine Decay: The step size shrinks smoothly and gradually, like sliding down a smooth curve, getting very small near the end.
- Step Decay: The step size stays big for a while, then suddenly drops by half (or some other factor), stays there for a while, then drops again. It looks like a staircase going down.

If you watch the graph of your altitude (loss) over time, it usually goes down sharply at first, then flattens out as you approach the bottom. If it goes up, you took a step that was too large and crashed into a wall. If it stays flat for too long, you are stuck in a plateau or your steps have become too small to matter.


## An analogy

Think of training a model like driving a car down a steep, winding mountain road while looking through a tiny window in the dashboard.

You want to reach the bottom of the mountain (zero error). You can only see the patch of road directly in front of your bumper. This visible patch represents the 'gradient'—the immediate slope of the road right where you are. If the road curves left, you turn the wheel left. The amount you turn the wheel is your 'learning rate.'

Early in the journey, the mountain is steep. You can push the accelerator hard because the downward slope guides you safely. But as you get closer to the bottom of the valley, the road flattens out. If you keep pressing the accelerator (high learning rate), you will zoom past the target location and crash into the buildings on the other side. This is why we use a 'schedule': we gradually ease off the gas pedal. 

'Cosine decay' is like smoothly rolling your foot off the gas as you approach the destination. 'Step decay' is like hitting the brakes hard every few miles, then easing up again.

A 'plateau' is like driving onto a long, flat plateau midway down the mountain. You are moving forward, but you aren't going down. The road feels level. If you don't notice this change in the slope (the gradient becoming near-zero), you will drive across the entire plateau without realizing you haven't reached the bottom.

This analogy has limits. In a car, the road is fixed. In training, the 'road' changes as you learn because the data you are processing is sampled randomly. Also, the mountain has thousands of dimensions, not just height and direction, so you can never see the whole landscape at once. But the core idea holds: you are navigating based on immediate local information, using rules to adjust how aggressively you move.

One caveat: unlike a car, if your step is too big in training, you don't just crash; the math can break entirely, producing errors like 'infinity' or 'NaN' (not a number), which stop the process completely.


## Definition

Training dynamics is the observable behavior of how a model's error and internal adjustments change over time during learning, specifically governed by the direction of steepest descent and the planned size of each update step.

In simpler terms: It is the study of how fast and in what pattern a model improves, crashes, or stalls as it learns from data.


## How it connects

This concept sits directly beside **Gradient Descent**, which is the specific mathematical recipe used to calculate the direction of the slope (gradient) at each step. Training dynamics is the result you see when you run Gradient Descent over time.

It also connects closely to **Learning Rate Scheduling**, which is the practice of defining the rules for how the step size changes during training (such as using cosine decay or step decay). Understanding dynamics helps you choose the right schedule for your specific problem.


## The mathematics

First, let's name the branch of mathematics this concept draws on: it is called **calculus**, specifically the part about **derivatives**. Calculus is the mathematics of change — how a quantity changes when another quantity changes. You know arithmetic (adding, multiplying) and algebra (using letters for numbers). Calculus adds one new idea: the **slope**. Slope is a measure of steepness. In everyday life, if you walk up a hill, the slope tells you how much your altitude changes for each step you take forward. If the ground is flat, the slope is zero. If it's steep, the slope is a big number. In training, the 'hill' is the loss — a number that tells us how wrong the model is. The 'position' on the hill is the set of weights. The slope of the loss with respect to a weight is called the **gradient**. The gradient tells us: if we nudge this weight up a tiny bit, how much will the loss change? If the gradient is positive, then increasing the weight makes the loss go up, so we should decrease the weight. If the gradient is negative, increasing the weight makes the loss go down, so we should increase it. Training takes the gradient, flips its sign (because we want to go downhill), and moves the weights in that direction by a small step called the **learning rate**. The learning rate is a single number that controls how big the step is — too big and you overshoot, too small and you crawl. The **loss curve** is a graph of the loss value over time (each time step is called an **epoch** or an **iteration**). Its shape — whether it goes down smoothly, spikes, or flattens — tells us about the training. **Gradient norms** are the lengths of the gradient vectors (we'll get to what a vector is) — they tell us the overall steepness. If they're very large or very small, that signals dangerous conditions. **Learning rate schedules** are rules for changing the learning rate over time, because what works at the start of training (big steps to make quick progress) doesn't work at the end (small steps to fine-tune). The whole process is called **iterative descent** — you repeat the step downhill many times, always getting closer (usually) to a minimum. Now let's make this concrete with numbers.

The update rule for the weight w is: w_new = w_old - learning_rate * gradient. The gradient is the derivative of the loss with respect to w, written as d(loss)/dw. For the squared loss (w - 3)^2, the gradient is 2*(w - 3). More generally, the loss is a function L(w) of many weights, and the gradient is a vector of all the partial derivatives, written ∇L(w). The learning rate is often written as η (eta). So the full update: w ← w - η ∇L(w). In words: subtract the learning rate times the gradient from the current weights.


## A worked example

``` 
Imagine a model with just one weight, call it w. The loss function is the squared error: loss = (w - 3)^2. The goal is to find w that makes loss as small as possible. Clearly, when w = 3, loss = 0. We'll start at w = 0 and use a learning rate of 0.1. We'll do a few steps to see the dynamics.

Step 1: At w=0: loss = (0 - 3)^2 = 9. The gradient (slope) is 2*(0-3) = -6.   # The slope of (w-3)^2 is 2*(w-3). At w=0, slope = -6. Since the slope is negative, increasing w will decrease the loss, so we should increase w.
Step 2: Update: w = 0 - 0.1 * (-6) = 0.6.   # We move w in the opposite direction of the gradient, scaled by the learning rate 0.1. So w increases by 0.6.
Step 3: At w=0.6: loss = (0.6-3)^2 = 5.76. Gradient = 2*(0.6-3) = -4.8.   # The loss went down from 9 to 5.76, and the gradient got less negative.
Step 4: Update: w = 0.6 - 0.1 * (-4.8) = 1.08.   # Again, move against the gradient.
Step 5: At w=1.08: loss = (1.08-3)^2 = 3.6864. Gradient = 2*(1.08-3) = -3.84.   # We're getting closer to 3, loss is dropping.
Step 6: Update: w = 1.08 - 0.1 * (-3.84) = 1.464.   # Continue the process.

Result: After these three steps, w has moved from 0 to 1.464, and loss has dropped from 9 to 3.6864. We're on the way to w=3. Notice that each step got a bit smaller (0.6, 0.48, 0.384) because the gradient is shrinking as we approach the minimum. This is a natural deceleration — training gradually slows down near the bottom.
```


## From scratch, in Python

```python
# We'll simulate the one-weight example from the maths section.
# No external libraries, just plain Python.

# We need a way to repeat steps. That's a for loop.
# A for loop repeats the indented block a fixed number of times.

# Start with the initial weight and a small learning rate.
w = 0.0
learning_rate = 0.1

# We'll store the loss history in a list.
# A list is a container that keeps items in order. We'll use it to plot later.
loss_history = []

# Repeat 10 times.
for step in range(10):
    # Compute the loss at the current weight.
    loss = (w - 3) ** 2
    # Compute the gradient (derivative of loss w.r.t. w).
    gradient = 2 * (w - 3)
    # Update the weight: move against the gradient.
    w = w - learning_rate * gradient
    # Record the loss before the update (we want the loss at the start of each step).
    loss_history.append(loss)

# Print the last few losses to see how they change.
print("Last 4 losses:", loss_history[-4:])
print("Final weight:", w)

```


## The same thing, with a framework

```python
import numpy as np

# In NumPy, we can vectorize the operation for many weights at once.
# Here we're still just doing one weight, but the syntax is the same.
w = np.array([0.0])
learning_rate = 0.1

for step in range(10):
    loss = (w - 3) ** 2
    gradient = 2 * (w - 3)
    w = w - learning_rate * gradient
    print(f"Step {step}: loss={loss[0]:.4f}, w={w[0]:.4f}")

```


## What the code does

The plain Python code does exactly the arithmetic we did by hand: for each of 10 steps, it calculates the loss (how far we are from 3, squared), calculates the gradient (2 times the difference), then updates the weight by subtracting the learning rate times the gradient. The `loss_history` list stores each loss so we could later plot the loss curve. The `for step in range(10):` loop runs the block 10 times. The output shows the loss decreasing from 9 to about 0.3, and the weight converging toward 3. The NumPy version does the same thing but uses `numpy.array` to represent the weight — in practice, models have thousands or millions of weights, and NumPy or PyTorch let you update them all at once in a single line.


## Common misconceptions

**Lower loss is always better** — If the training loss is going down, everything is fine.

Training loss can go down while validation loss goes up (overfitting). Also, a low training loss might happen in a sharp local minimum that doesn't generalize. In fact: Watch both training and validation loss. A diverging gap means overfitting. Also, the shape of the loss curve matters: a sharp drop followed by a plateau might indicate you've found a poor minimum.

**Learning rate is a fixed constant for the whole training** — You set the learning rate once and never change it.

A constant learning rate either wastes time early (if too small) or fails to fine-tune late (if too large). Schedules that decay the learning rate are standard in practice. In fact: Use a schedule: start with a warmup, then decay (step, cosine, or linear). The schedule is part of the training recipe.

**The loss curve is smooth** — Training loss should go down smoothly without any bumps.

In stochastic training, randomness in batches causes noisy gradients, so the loss curve jitters. Sharp spikes can happen but are often a sign of a learning rate that's too high. In fact: Expect noisy but generally downward trend. Smoothness suggests a full-batch gradient (rare). Use the noise as a diagnostic.

**A plateau means the model is done learning** — If the loss stops decreasing for a while, training is complete.

Plateaus can be temporary — the model might cross flat regions and then descend sharply. Also, a plateau might mean the learning rate is too small to escape a shallow local minimum. In fact: Try increasing the learning rate or using momentum. Watch for sudden drops after a plateau, which indicate a new region was found.

**Gradient norm is just a number, not important** — The gradient norm is an obscure side detail.

Gradient norms reveal the scale of updates. If they are tiny, the learning rate is effectively too small; if huge, the update will blow up. In fact: Monitor gradient norms; if they explode, use gradient clipping. They often warn about issues before the loss shows them.


## In short

Training dynamics is the study of how loss, gradients, and learning rate change over time. The learning rate controls step size — too big diverges, too small crawls. Learning rate schedules (warmup, decay) are essential for efficient and stable training. The loss curve's shape diagnoses problems: spikes mean too high a rate, plateaus mean stuck, and smooth descents mean all is well. Gradient norms tell you about scale before the loss does. In practice, every serious training run uses a schedule and monitors these signals.


## Where this is used

In **large language models (LLMs)** like GPT-4, training dynamics determines whether a multi-billion-parameter model converges or diverges. The loss curve is watched in real time by engineers; a sudden spike often means the learning rate is too high, and they'll reduce it or restart from a checkpoint. Learning rate schedules are essential: most LLMs use a **warmup** phase where the learning rate ramps up from near zero to a peak over a few thousand steps, then decays following a cosine curve. This schedule is so important that a poor choice can waste an entire training run worth millions of dollars. Gradient norms are also monitored: if they explode, the update will blow up the weights; this is why **gradient clipping** is used. In **small models running on devices** (like phone keyboards predicting the next word), training dynamics is used to fine-tune a small model on-device with limited data. The loss curve might plateau quickly because the model has limited capacity; a schedule that decays the learning rate helps it settle into a good solution without overfitting. In **retrieval-augmented generation (RAG)**, the embedding models (which convert text to vectors) are trained with contrastive loss, and their training dynamics show sharp drops when the model first learns to separate similar from dissimilar pairs. A developer might use a warmup schedule to stabilize training before the loss falls. In **agentic systems** that plan and call tools, the underlying policy model is often fine-tuned (e.g., with RLHF) — training dynamics here involves reward curves that can be noisy and prone to spikes; practitioners use small learning rates and monitor the reward curve to avoid catastrophic forgetting of the base language model. In **everyday applied machine learning** — like fraud detection or recommendation systems — data scientists train gradient-boosted trees or neural nets and always plot the training and validation loss curves. If they see a plateau early, they increase the learning rate or add more capacity. If the loss diverges, they reduce the learning rate. They also use learning rate schedules like step decay (e.g., reduce the rate by half every 10 epochs) to fine-tune. A developer typically writes code in PyTorch like this: `optimizer = torch.optim.SGD(model.parameters(), lr=0.1)` and then each epoch calls `optimizer.step()`. To implement a schedule, they use `torch.optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.5)`. If done badly — say, a learning rate that's too high — the loss curve will show oscillation or NaN values, and the model is useless. Done well, the loss decreases smoothly, and the model achieves good accuracy.


---

Read this on the web: https://basavapurushottam.com/learn-ai/training-dynamics
