# Cross-Entropy Loss

The negative log-likelihood of the correct class under the model's predicted distribution — the standard objective for classification and next-token prediction.

**Rests on:** KL Divergence, Shannon Entropy

**Leads to:** Cross-Lingual Transfer, Double Descent, Logistic Regression, Rare Event Prediction

*Mathematical Foundations · Information Theory*


## Why learn this

Cross-entropy loss is the mechanism that teaches computer programs to classify things, from sorting your email into spam or not-spam to deciding which word comes next in a sentence generated by an AI. When a program guesses the answer to a multiple-choice question, it gives a percentage chance for every possible option. Cross-entropy takes those percentages and calculates a single score that tells you exactly how wrong the guess was. This score is what the program uses to adjust its brain so it makes fewer mistakes next time. Understanding this helps you see why some AI models are confidently incorrect about simple facts, and why this specific method of measuring error is the standard for almost all modern pattern-recognition tasks.


## The intuition

Imagine a test where a computer has to guess whether an image shows a cat or a dog. It doesn't just shout 'cat'; it says 'I am 90% sure this is a cat, and 10% sure it is a dog.' If the image actually was a cat, we look at that 90% figure. In cross-entropy loss, we ask: how much surprise should we have? If the computer had put 90% probability on the correct answer (cat), the loss is very low—about 0.1. This means it was barely surprised because it got it right.

But imagine the same cat image, and the computer says 'I am only 1% sure this is a cat.' The loss here jumps to about 2.3. That number is much higher, reflecting that the computer was very wrong. The formula uses a mathematical trick called the negative logarithm to turn these probabilities into loss scores. If the computer were completely wrong and gave the correct answer a probability of 0%, the loss would go to infinity. This ensures the system learns aggressively to avoid being confidently wrong.

Consider two specific guesses for a horse race:
1. The true winner is Horse A. The model gives Horse A a 50% chance. The loss is approximately 0.69.
2. The true winner is Horse A. The model gives Horse A a 1% chance. The loss is approximately 4.60.
The difference between 0.69 and 4.60 shows how much more punishing it is to be confidently wrong than to be uncertain.


## An analogy

Think of a weather forecaster who must bet on tomorrow's weather each day. Every morning, she announces probabilities for the possible outcomes: '70% sun, 20% rain, 10% snow.' She doesn't just guess 'sun'; she assigns a spread of confidence across all options. Her performance isn't judged by whether she guessed the single outcome correctly, but by how well her predicted spread matches reality.

If it rains tomorrow, and she had assigned only a 1% chance to rain, she has bet against the event that happened. The penalty for this is huge because she was very confident in her wrongness. If she had assigned a 90% chance to sun (when it rained), the penalty is large, but if she had said '30% sun, 30% rain, 40% snow,' the penalty is smaller because she wasn't surprised by the rain.

Cross-entropy loss is exactly this betting penalty. The forecaster's predicted percentages are the model's outputs; the actual weather is the true label. The loss is calculated from the negative logarithm of the probability she assigned to what actually happened. A forecaster who always predicts '100% sun' will have a very low average penalty on sunny days, but if one rainy day occurs, the loss becomes infinite. This forces her to spread her bets realistically. The analogy breaks down in one small way: a human might feel bad about being wrong, but the model only cares about minimizing the mathematical penalty over many days.


## Definition

Cross-entropy loss is a single number that measures how poorly a predicted set of probabilities matches the actual outcome, calculated by taking the negative logarithm of the probability assigned to the correct option and averaging this over all examples.


## How it connects

This concept sits directly beside two neighboring ideas. First is the technique for converting raw numerical scores into probabilities that add up to 100%, which ensures every class gets a fair chance to be selected. Second is Shannon entropy, which measures the inherent unpredictability of a specific situation—like how hard it is to guess the weather in London compared to the desert. Cross-entropy tells you the total error of your prediction, while entropy tells you the baseline difficulty of the task itself.


## The mathematics

Cross-entropy loss is a way to measure how wrong a model's predictions are when it tries to guess which category something belongs to, like whether an image shows a cat or a dog.

The idea starts with likelihood. Likelihood asks: if my model says there is a 90% chance this is a cat, and it actually is a cat, how good was that guess? The answer is the number 0.9. If the model said there was only a 10% chance, but it was still a cat, the likelihood is 0.1. A higher number means the model was more confident and correct.

When we have many examples to check, we multiply their individual likelihoods together. For example, if two examples both had a likelihood of 0.9, the total likelihood is 0.9 times 0.9, which equals 0.81. But as we add more examples, this product gets smaller and smaller, eventually becoming so tiny that computers struggle to handle it accurately. To fix this, we use a mathematical trick: we take the logarithm (often called 'ln' or natural log) of each likelihood instead.

The logarithm has a special property: it turns multiplication into addition. Instead of multiplying 0.9 by 0.9, we add ln(0.9) to ln(0.9). This keeps the numbers at a manageable size. However, because probabilities are between 0 and 1, their logarithms are negative numbers. In mathematics, we usually want our 'loss' or 'error' to be a positive number that we try to make as small as possible. So, we take the negative of the logarithm. This flips the sign, turning -0.1 into +0.1.

This specific calculation is called cross-entropy loss. For one single example, the formula is:

Loss = - ln(p)

Here, 'p' is the probability the model assigned to the correct class. The symbol 'ln' means the natural logarithm. Because p is less than 1, ln(p) is negative, so -ln(p) is positive.

For a whole dataset with N examples, we calculate this loss for every example and then take the average. If y_i is the true class for example i, and p[i, y_i] is the probability the model gave to that true class, the total cross-entropy loss is:

Total Loss = -(1 / N) * [ ln(p[1, y_1]) + ln(p[2, y_2]) + ... + ln(p[N, y_N]) ]

In plain words: add up the negative logarithms of the probabilities for the correct answers, then divide by the number of examples.

Why is this loss function useful? It punishes wrong predictions with high confidence very harshly. Imagine the true class is 'cat'.

Case A: The model predicts 0.9 probability for cat. 
ln(0.9) is approximately -0.105. 
The loss is -(-0.105) = 0.105.

Case B: The model predicts 0.1 probability for cat (it is very confident it is NOT a cat). 
ln(0.1) is approximately -2.303. 
The loss is -(-2.303) = 2.303.

The error in Case B is more than twenty times larger than in Case A. This tells the learning algorithm to adjust its weights much more aggressively when it is confidently wrong, which helps it learn faster.


## A worked example

Let us walk through a concrete example with real numbers.

Scenario: A model classifies images into three classes: Cat (class 1), Dog (class 2), and Bird (class 3).

We look at one single image. The true label is 'Cat' (class 1).
The model outputs the following probabilities for each class:
- Probability of Cat: 0.7
- Probability of Dog: 0.2
- Probability of Bird: 0.1

Note that these add up to 1.0 (0.7 + 0.2 + 0.1 = 1.0), which is required for a valid probability distribution.

Step 1: Identify the probability assigned to the correct class.
The true class is Cat. The model gave Cat a probability of 0.7.
So, p = 0.7.

Step 2: Calculate the natural logarithm of this probability.
We need ln(0.7).
Using a calculator or table, ln(0.7) ≈ -0.3567.
(This number is negative because 0.7 is less than 1.)

Step 3: Apply the negative sign to get the loss.
Loss = - ln(p)
Loss = - (-0.3567)
Loss = 0.3567

Result: The cross-entropy loss for this single example is approximately 0.357.

Comparison with other predictions:
If the model had been more confident and correct, predicting p = 0.9 for Cat:
ln(0.9) ≈ -0.1054
Loss = -(-0.1054) = 0.105
The loss is lower (better) because the prediction was closer to 1.

If the model had been confident but wrong, predicting p = 0.1 for Cat:
ln(0.1) ≈ -2.3026
Loss = -(-2.3026) = 2.303
The loss is much higher (worse) because the model was confidently incorrect.


## From scratch, in Python

```python
import math

# Example: three classes, one true label (class 1)
probs = [0.7, 0.2, 0.1]
true_class = 1  # cat

# We only need the probability of the true class
p_true = probs[true_class - 1]  # indexing starts at 0, so subtract 1

# Cross-entropy loss for one example
loss = -math.log(p_true)
print(loss)  # 0.35667494393873245

# For a batch of examples, average the losses
batch_probs = [
    [0.7, 0.2, 0.1],  # example 1, true class 1
    [0.1, 0.8, 0.1],  # example 2, true class 2
    [0.2, 0.2, 0.6],  # example 3, true class 3
]
batch_true = [1, 2, 3]

losses = []
for row, t in zip(batch_probs, batch_true):
    p = row[t - 1]
    losses.append(-math.log(p))

average_loss = sum(losses) / len(losses)
print(average_loss)  # 0.4457789092454983
```


## The same thing, with a framework

```python
import torch
import torch.nn.functional as F

# Example: logits (raw scores) for 3 classes, batch of 2
logits = torch.tensor([[2.0, 1.0, 0.5], [0.2, 0.8, 1.5]])
targets = torch.tensor([0, 2])  # true class indices

# PyTorch's cross_entropy combines softmax and loss in one step
loss = F.cross_entropy(logits, targets)
print(loss)  # tensor(0.8268)
```


## What the code does

The from-scratch code has three parts. First, we import math to get the logarithm. Then we define the model's probabilities for one example — a list of three numbers — and pick the true class. Since Python lists are indexed from 0, class 1 is at position 0, so we subtract 1. We pull out the probability for the true class and take the negative logarithm. For a batch, we use a list of lists — each inner list is one example's probabilities — and a matching list of true class indices. A for loop goes through them in pairs, computes the loss for each, and we average. The framework version uses PyTorch's built-in cross_entropy, which takes raw logits and the true class indices, and applies softmax internally — so you don't have to do it yourself. It also uses the log-sum-exp trick for numerical stability, which is why it's preferred in practice.


## Common misconceptions

**Cross-entropy is just a 'correct vs wrong' count** — The loss doesn't care how confident I am — it only cares if I got the right answer.

A model that is 99% confident on the true class has a loss of 0.01, while one that is 51% confident has a loss of 0.67. The difference is huge, even though both are 'right'. The loss heavily penalises being wrong with high confidence. In fact: Cross-entropy quantifies the confidence-adjusted error — being right but hesitant is penalised, being wrong but confident is punished severely.

**Zero loss means perfect** — If my loss is zero, the model is perfect and can't be improved.

Zero loss means the model assigned probability 1.0 to every correct answer, which on real data is impossible because data has noise and overlap. It also often means the model is overfitting, memorising the training set. In fact: Loss close to zero is a red flag for overfitting. On real data, a small but non-zero loss is the goal.

**Any probability can be used** — I can feed raw scores into cross-entropy; they're kind of like probabilities.

Raw scores (logits) can be negative, bigger than 1, and don't sum to 1. Taking the log of a negative number is undefined; the loss becomes nonsense. In fact: You must convert logits to probabilities with softmax (which squashes them to sum to 1) before computing cross-entropy. Many libraries combine this step.

**Cross-entropy is symmetric** — If I swap the predicted and true distributions, the loss is the same.

In the formula, you take the log of the predicted probability. Swapping gives a different value, just like a forecaster who says '90% rain' when it's sunny is different from a forecaster who says '10% rain' when it's sunny. In fact: Cross-entropy is asymmetric — the true label is fixed, and you measure how the predicted distribution aligns with it.

**Lower loss is always better** — My only goal is to make the loss as small as possible, no matter what.

Pushing loss to zero can lead to overfitting, where the model memorises the training data and fails on new data. There's a balance between fitting and generalising. In fact: You want the loss to be low on *unseen* data, not just the training set. Monitoring validation loss is crucial.


## In short

Cross-entropy loss = negative log of the probability assigned to the true class. It's the standard for classification and language modelling because it rewards correct confidence and harshly punishes confident mistakes. It pairs with softmax, gives a clean gradient, and appears everywhere from spam filters to GPT. Remember: it's a measure of surprise — the less surprised you are by the right answer, the lower the loss.


## Where this is used

Cross-entropy loss is everywhere in modern AI because classification is everywhere. In large language models, it's the sole training objective: the model predicts a probability over the entire vocabulary, and cross-entropy measures how well it predicts the next token. During training, billion-parameter models compute this for every token in the training corpus, millions of times. The loss is what backpropagation uses to adjust the weights — a lower loss means better next-token prediction, which is what makes the model coherent. In small models running on devices, cross-entropy is still used, but the vocabulary is smaller and the computations are done with quantized weights — the loss is the same, just computed more efficiently. For retrieval-augmented generation, cross-entropy isn't directly in the retrieval step, but the embedding model that retrieves documents is trained with cross-entropy over which document is relevant. In agents that plan and call tools, cross-entropy loss is used to train the model to select the right tool call from a set of options — the same classification setup. In everyday applied work, cross-entropy is the default for any classification task: spam detection, image recognition, fraud detection, and speech recognition. When a developer trains a model, they write code like `loss = F.cross_entropy(logits, targets)` in PyTorch, or `model.compile(loss='categorical_crossentropy')` in Keras. What breaks when done badly: if you use square error instead of cross-entropy, the model learns slowly and often gets stuck; if you forget to use softmax and treat raw scores as probabilities, the loss becomes meaningless; if you use cross-entropy on a task that isn't classification — like regression — it fails entirely. As your notes from the Mathematics Behind Neural Networks say, cross-entropy with softmax gives a clean gradient: the gradient is simply the difference between the predicted probability and the true one-hot vector, which is why training is so much smoother.


---

Read this on the web: https://basavapurushottam.com/learn-ai/cross-entropy-loss
