# Loss Function Selection

Choosing the objective that encodes what counts as a good prediction — squared error, absolute error, cross-entropy or a task-specific alternative.

**Rests on:** Least Squares Regression, Logistic Regression

*Deep Learning · Deep Learning*


## Why learn this

Choosing how to measure a mistake is the most important design choice in building an AI model because it dictates what the model prioritizes. Every model learns by trying to reduce this 'cost' score. If you define the cost incorrectly, the model will become very good at optimizing the wrong goal. Understanding this allows you to fix models that are failing in specific ways—for example, a model that is afraid to make big mistakes versus one that ignores small errors. This concept is the foundation for how any learning system improves over time.


## The intuition

Imagine you are a teacher grading an exam where each student's answer is a number, such as a forecast of tomorrow's temperature. The true temperature is 30°C. Student A predicts 29°C, making an error of 1 degree. Student B predicts 20°C, making an error of 10 degrees. You need a single number that says how bad each prediction is. Any sensible rule gives a higher number for B than for A, but the rules differ in how much they punish the error.


## An analogy

Think of choosing a loss function like choosing a measuring stick for a tailor making suits. Your customer wants a suit that fits. You can measure the fit in two ways: by the total length of cloth that hangs off (too big) or is missing (too tight), or by the square of that overhang. With a simple ruler (absolute error), a suit that is 2 cm too big in the shoulders and 2 cm too tight in the waist scores 4. With a squared ruler, each 2 cm deviation becomes 4, so the total is 8. Now imagine a suit that is 10 cm too big in one place and perfect everywhere else: the simple ruler scores 10, the squared ruler scores 100. The squared ruler makes that one glaring mistake look catastrophic — so a tailor told to minimise squared error will obsess over the worst-fitting part, even if it means making several other parts slightly off. The simple ruler treats all mistakes equally, so the tailor might accept one big mistake if it fixes many small ones. Neither is universally 'right' — it depends on what you, the customer, care about. If a single glaring error makes the suit unwearable, choose squared. If many small errors are as annoying as one big one, choose absolute. Real-world losses (like Huber) are compromises: they behave like squared error for small errors and like absolute error for large ones. The analogy breaks down because in AI, you don't just measure — you adjust thousands of parameters to reduce the loss, and the shape of the loss function (flat, steep, with bumps) determines how easy that adjustment is. A tailor can only adjust seams; a model adjusts every weight.


## Definition

A loss function is a mathematical rule that takes a model's prediction and the true answer, and returns a single number — the cost — that says how wrong the prediction was; the model is trained by minimising this number.


## How it connects

You have already mastered the concept of 'activation functions' — the nonlinearity that gives a neuron its output. The loss function sits right after the activation: the activation produces the prediction, and the loss measures how far that prediction is from the truth. You have also seen 'gradient descent' — the process of nudging weights to reduce the loss. Without a loss function, gradient descent has nothing to aim at. In your library notes, you have 'Deep Learning From Scratch with Python (O'Reilly ed.)', which says: 'Next-token prediction is multi-class classification over the vocabulary, which is why cross-entropy is the natural loss.' This page builds the foundation for that claim — you will see why cross-entropy suits classification, and why squared error would be a poor choice for predicting which word comes next. Your notes on 'adaptive filtering' use squared error as the default loss; you will see here why that is a reasonable default for signal tracking, and when it fails.


## The mathematics

The mathematics behind choosing a loss function is simple arithmetic: addition, subtraction, multiplication, and squaring. You do not need calculus to understand what a loss function *is*. Calculus is only needed later to find the best numbers for your model. The loss function is just a rule that takes two numbers—the model's prediction and the true answer—and turns them into one number called the 'cost' or 'loss'.

The most common choice is **squared error**. Here is how it works step by step:
1. Take the prediction and subtract the true value. This gives you the 'error'.
2. Multiply that error by itself (square it). This makes sure the cost is always positive, so a negative error (under-prediction) costs just as much as a positive error (over-prediction) of the same size.
3. Do this for every example in your data set.
4. Add all those squared errors together.

For example, if you predict 5 and the truth is 3, the error is 2. Squared, that is 4. If you predict 3 and the truth is 5, the error is -2. Squared, that is also 4. The formula for the average squared error (often called Mean Squared Error or MSE) is:

Sum of (true value minus predicted value) squared, divided by the number of examples.

Another choice is **absolute error**. This is simpler:
1. Take the prediction and subtract the true value.
2. Ignore whether the result is positive or negative. Just take its size (its absolute value).
3. Add up these sizes for all examples.

Absolute error treats a small error and a large error more linearly. Squared error makes large errors much worse than small ones because squaring amplifies them.

A third choice, used when predicting categories (like 'cat' or 'dog'), is **cross-entropy**. This measures how 'surprised' the model is by the truth. If the model predicts a 90% chance for the correct answer, the cost is low. If it predicts a 10% chance for the correct answer, the cost is high. The formula involves logarithms, which are just a way to compress large ranges of numbers. In short: cross-entropy punishes being confidently wrong much more than being confidently right rewards you.

The key mathematical insight is this: the loss function is not forced on you by the data. It is a tool you choose. If you care about avoiding huge mistakes, use squared error. If you want to be robust against outliers (extreme values), use absolute error or Huber loss. The choice changes how the model learns.


## A worked example

Let us calculate three different losses for three days of temperature forecasts.

**The Data:**
- Day 1: True value = 30, Prediction = 29
- Day 2: True value = 25, Prediction = 22
- Day 3: True value = 20, Prediction = 18

**Step 1: Calculate the raw errors (Prediction minus Truth)**
- Day 1 error: 29 - 30 = -1
- Day 2 error: 22 - 25 = -3
- Day 3 error: 18 - 20 = -2

**Step 2: Calculate Squared Error**
We square each error to remove the negative sign and amplify larger mistakes.
- Day 1 squared error: (-1) * (-1) = 1
- Day 2 squared error: (-3) * (-3) = 9
- Day 3 squared error: (-2) * (-2) = 4

Total Sum of Squared Errors = 1 + 9 + 4 = 14.
If we want the Mean Squared Error (average), we divide by the number of days (3): 14 / 3 ≈ 4.67.
Notice how Day 2's error of 3 became a cost of 9. The large mistake is weighted heavily.

**Step 3: Calculate Absolute Error**
We take the size of each error, ignoring the sign.
- Day 1 absolute error: | -1 | = 1
- Day 2 absolute error: | -3 | = 3
- Day 3 absolute error: | -2 | = 2

Total Sum of Absolute Errors = 1 + 3 + 2 = 6.
Here, the cost is proportional to the size of the mistake. A mistake of 3 costs 3 times as much as a mistake of 1.

**Step 4: Calculate Huber Loss (with threshold delta = 1)**
Huber loss is a hybrid. It uses squared error for small mistakes (to be smooth) and absolute error for large mistakes (to avoid extreme penalties). We use a threshold, called delta (δ), set to 1.
Rule: If the size of the error is less than or equal to 1, use half the squared error. If it is greater than 1, use: delta * (size of error - half of delta).

- Day 1: Error size is 1. This is not greater than 1, so we use the squared part.
  Cost = 0.5 * (-1)^2 = 0.5 * 1 = 0.5.

- Day 2: Error size is 3. This is greater than 1, so we use the linear part.
  Cost = 1 * (3 - 0.5 * 1) = 1 * (3 - 0.5) = 1 * 2.5 = 2.5.

- Day 3: Error size is 2. This is greater than 1, so we use the linear part.
  Cost = 1 * (2 - 0.5 * 1) = 1 * (2 - 0.5) = 1 * 1.5 = 1.5.

Total Huber Loss = 0.5 + 2.5 + 1.5 = 4.5.

**Comparison:**
- Squared Error Total: 14
- Absolute Error Total: 6
- Huber Loss Total: 4.5

All three methods agree that Day 2 was the worst day. However, Squared Error makes Day 2's mistake look 9 times worse than Day 1's. Huber loss limits this amplification, making it only 2.5 times worse. This shows how the choice of loss function changes which errors dominate the training process.


## From scratch, in Python

```python
# Loss functions from scratch, using only plain Python.
# We'll use three lists: true values, predicted values.

# Step 1: The building blocks.
# A list is an ordered collection of values. It keeps its order and can be changed.
true_temps = [30, 25, 20]
pred_temps = [29, 22, 18]

# A for loop lets you go through each item in a list one at a time.
# Here we print each error.
print("Errors:")
for t, p in zip(true_temps, pred_temps):
    error = p - t
    print(f"true {t}, predicted {p}, error {error}")
# Output:
# true 30, predicted 29, error -1
# true 25, predicted 22, error -3
# true 20, predicted 18, error -2

# Step 2: A function is a reusable block of code. It takes inputs, does something, and returns a result.
def mean_squared_error(true_values, predicted_values):
    total = 0
    n = len(true_values)  # len() gives the number of items in a list
    for t, p in zip(true_values, predicted_values):
        error = p - t
        total = total + (error ** 2)  # ** means 'to the power of' — error squared
    return total / n  # divide by the number of examples to get the mean

def mean_absolute_error(true_values, predicted_values):
    total = 0
    n = len(true_values)
    for t, p in zip(true_values, predicted_values):
        error = p - t
        total = total + abs(error)  # abs() gives the absolute value
    return total / n

def huber_loss(true_values, predicted_values, delta=1.0):
    total = 0
    n = len(true_values)
    for t, p in zip(true_values, predicted_values):
        error = p - t
        abs_error = abs(error)
        if abs_error <= delta:
            total = total + 0.5 * (error ** 2)
        else:
            total = total + delta * (abs_error - 0.5 * delta)
    return total / n

# Now compute and print the losses.
print("MSE:", mean_squared_error(true_temps, pred_temps))
print("MAE:", mean_absolute_error(true_temps, pred_temps))
print("Huber (delta=1):", huber_loss(true_temps, pred_temps))
# Output:
# MSE: 4.666666666666667
# MAE: 2.0
# Huber (delta=1): 1.5

# Cross-entropy for classification.
# true_probs: a list of one-hot vectors, e.g., [1,0,0] means class 0 is true.
# pred_probs: a list of softmax outputs, e.g., [0.7,0.2,0.1].
# We'll use plain Python lists of lists.
true_one_hot = [1, 0, 0]  # true class is index 0
pred_probs = [0.7, 0.2, 0.1]  # predicted probabilities for each class

import math  # we need the log function

# Cross-entropy for a single example:
loss = 0
for t, p in zip(true_one_hot, pred_probs):
    if t == 1:
        loss = loss + math.log(p)  # log of the predicted probability of the true class
# Note: we do not add if t==0, because 0*log(p) is 0.
# The loss is the negative of that sum (because log(p) is negative when p<1).
loss = -loss
print("Cross-entropy loss:", loss)  # Output: Cross-entropy loss: 0.35667494393873245

# Let's see what happens when the model is confident but wrong:
true_one_hot_wrong = [1, 0, 0]
pred_probs_wrong = [0.1, 0.8, 0.1]  # predicted only 0.1 for the true class
loss_wrong = 0
for t, p in zip(true_one_hot_wrong, pred_probs_wrong):
    if t == 1:
        loss_wrong = loss_wrong + math.log(p)
loss_wrong = -loss_wrong
print("Cross-entropy (confident wrong):", loss_wrong)  # Output: ~2.302585092994046

# That is much larger — it punishes overconfident wrong predictions heavily.
```


## The same thing, with a framework

In practice, you would use NumPy or PyTorch. NumPy computes the same things in a few lines:

```python
import numpy as np
true = np.array([30, 25, 20])
pred = np.array([29, 22, 18])

mse = np.mean((true - pred) ** 2)
mae = np.mean(np.abs(true - pred))

# Huber: use a library or implement with np.where
delta = 1.0
errors = np.abs(true - pred)
huber = np.where(errors <= delta, 0.5 * (true-pred)**2, delta * (errors - 0.5*delta)).mean()

# Cross-entropy: for a single example with one-hot true and softmax output
true_one_hot = np.array([1.0, 0.0, 0.0])
pred_probs = np.array([0.7, 0.2, 0.1])
ce = -np.sum(true_one_hot * np.log(pred_probs))
```

What this does underneath: `(true - pred)` is the element-wise subtraction — NumPy broadcasts the subtraction across the arrays, giving `[1, 3, 2]`. Squaring is element-wise too, and `.mean()` sums and divides by the number of elements. The `np.where` is a vectorised conditional — it applies the threshold to every element at once, exactly like the `if` statement in the plain Python loop. The cross-entropy line multiplies the one-hot vector element-wise by the log of the predictions, then sums — that is precisely the loop we wrote earlier, but expressed as array operations. In PyTorch, you would typically use `torch.nn.MSELoss()` or `torch.nn.CrossEntropyLoss()`, which are pre-built and also handle numerical stability internally — but now you know exactly what they compute.


## What the code does

We start by defining two lists: `true_temps` and `pred_temps`. In Python, a list is written with square brackets: `[30, 25, 20]` is a list of three numbers. The `zip` function pairs up the first item of each list, then the second item, and so on — so in the first loop, `t` is 30 and `p` is 29. The `for` loop runs once for each pair. The `print(f"...")` line uses an f-string, which inserts variable values into the text. In the `mean_squared_error` function, we initialise `total` to 0, then for each pair we add the square of the error. `error ** 2` means 'raise to the power of 2', i.e., square it. After the loop, we divide by the length of the list — `len(true_values)` — to get the mean. This is exactly the mathematical formula: multiply out the differences, square them, sum, divide by n. The `mean_absolute_error` is identical except we use `abs(error)` — the absolute value — and do not square. The `huber_loss` function uses a conditional: `if abs_error <= delta:` — if the error is small, we use the squared formula; otherwise, we use the linear formula. This implements the hybrid we saw in the maths section. For cross-entropy, we start with a one-hot vector `[1,0,0]` — meaning the true class is index 0. The predicted probabilities are `[0.7,0.2,0.1]`. We loop over both, and only when `t==1` do we add `math.log(p)`. We then negate the sum, because `log(p)` is negative when p is less than 1. The result — 0.3567 — is the cross-entropy loss for this example. We then try a confident-wrong case: predicted 0.1 for the true class, and get 2.3026 — a much larger loss, showing the heavy penalty. The code is plain Python — no libraries except the standard `math` module.


## Common misconceptions

**There is one 'best' loss function** — I know the standard losses — MSE is for regression and cross-entropy is for classification. That's all I need.

The best loss depends entirely on your task and what you care about. MSE is great for balancing all errors, but it is lopsided: it treats a 10-unit error as 100 times worse than a 1-unit error. In many real tasks (like predicting delivery times), you'd rather have a few big errors than a systematic bias, and a different loss (or a custom one) serves you better. In fact: The loss function is a design choice. You pick the one that best matches your goal, and you can even design your own (asymmetric, weighted, or Huber-like) when standard ones misalign.

**Cross-entropy is only for classification with many classes** — Cross-entropy is overkill — I'll just use squared error on the probabilities.

Squared error on probabilities does not push the model to be confident. If the true class has probability 0.5, squared error gives a moderate cost whether the model is unsure or very wrong. Cross-entropy directly measures how wrong the probability assigned to the correct class is, and it grows without bound as the wrong class gets probability 1, which drives learning much more effectively. In fact: Cross-entropy is the natural loss for any probabilistic prediction, including single-label classification (a special case of multi-class). It directly optimises the likelihood of the true class.

**The loss function and the metric are the same** — If I train with MSE, I should evaluate with MSE, and that's fine.

The loss function is chosen to be smooth and differentiable so that gradient descent can work. Your business metric (like accuracy, F1, or revenue) may not be differentiable (e.g., accuracy is a step function), so you cannot train on it directly. You train on a proxy, but you evaluate on the real metric. In fact: You often train on a smooth surrogate loss (like cross-entropy) but evaluate on the true metric (like F1). This is a deliberate choice — the loss is the training objective, the metric is the evaluation objective.

**Absolute error is always worse than squared error** — Why would I ever use absolute error? Squared error gives a nicer curve for gradient descent.

Absolute error is not 'worse' — it is more robust to outliers. If one data point is far off, squared error will dominate the gradient and force the model to fix it, potentially at the expense of other points. Absolute error treats every point equally, so it is a better fit when outliers are not meaningful or are to be ignored. In fact: Use absolute error (or Huber) when you want to be robust to outliers; use squared error when you want to penalise large errors disproportionately (e.g., in fraud detection where a single huge case matters).

**Cross-entropy is always the right choice** — Cross-entropy is the modern standard, so I should use it everywhere — it's the safest bet.

Cross-entropy applies to probability predictions (classification, softmax outputs). For regression (predicting continuous numbers), it does not make sense — the output is not a distribution. Using it there would require converting to a distribution, and even then it may not match your goal. In fact: Match the loss to the task: cross-entropy for classification and probability modelling, squared/Huber for regression.

**The loss function is set once and never changes** — I pick a loss at the start and stick with it — changing it mid-training would confuse the model.

In practice, losses are often modified or combined. Curriculum learning changes the loss weighting over time; teacher-student distillation uses a weighted sum of hard and soft losses; and multi-task learning sums task-specific losses with learned weights. Changing the loss is a hyperparameter decision, not a sin. In fact: The loss can be dynamic: schedule the loss weight, combine losses, or switch to a different loss after a phase of training. As long as you know what you are optimising, it is a legitimate tool.


## In short

The loss function is how you tell a model what matters. For regression, choose mean squared error when you care about big errors disproportionately, mean absolute error when you want robustness to outliers, and Huber as a compromise. For classification and probability prediction, use cross-entropy — it directly rewards confidence in the correct class and punishes overconfidence in wrong ones. A mismatch between the loss and your business metric guarantees you optimise the wrong thing. In practice, developers often use a smooth surrogate loss for training and a separate metric for evaluation. Always ask: what do I actually want the model to be good at, and does this loss push it toward that?


## Where this is used

The choice of loss function is the single most important design decision in training any AI model, because it defines the objective the entire training process optimises. Get it wrong, and the model will cheerfully minimise a number that does not reflect what you actually care about.

**Large language models (LLMs).** The core training objective of every modern LLM — from GPT to LLaMA — is next-token prediction, and as your library notes from *Deep Learning From Scratch with Python* say, this is multi-class classification over the vocabulary, which is why **cross-entropy** is the natural loss. Concretely: the model outputs a probability distribution over, say, 100,000 possible next tokens; the true next token is one of them (a one-hot vector). Cross-entropy measures how much probability mass the model put on the true token — it is minimised when the model assigns probability 1 to the correct token. The mechanism: cross-entropy is differentiable and has a strong gradient even when the predicted probability is small, which helps learning. If you used squared error here, it would treat all wrong tokens equally — a prediction of 0.5 for the correct token and 0.5 for a random other token would get the same squared error as a confident 0.9 for the correct token — and would not push the model to concentrate probability on the true token. This is why cross-entropy is used universally for LLM training. The library says: 'Next-token prediction is multi-class classification over the vocabulary, which is why cross-entropy is the natural loss.' That is exactly right.

**Small language models (running on phones and laptops).** When a model must run on a device, the loss function is often *modified* to encourage efficiency. For example, some quantisation-aware training uses a **task-specific loss** (often cross-entropy for the task, plus a regularisation term that pushes weights toward values that quantise well). The cross-entropy is still the base, but a developer might add a penalty for large weight magnitudes, or use **knowledge distillation**: training the small model to match the output probabilities of a large teacher model, using a soft cross-entropy loss between the two distributions. The mechanism: the small model is trained to minimise the cross-entropy between its own output and the teacher's soft outputs (which carry more information than hard labels). This is why a small model can retain much of the large model's accuracy.

**Retrieval-Augmented Generation (RAG).** In RAG, a retrieval system finds relevant documents, and the generator (an LLM) reads them to answer a query. The loss functions appear in two places. First, the retriever itself is often trained with a **contrastive loss** (a form of cross-entropy): for a given query, the positive document (the one that answers it) should get a higher similarity score than negative documents. The loss is: −log( exp(sim(query, positive)) / Σ exp(sim(query, candidate)) ), which is exactly cross-entropy over a small set of candidates. Second, the generator still uses cross-entropy on the next token, but with the retrieved context prepended. The loss function choice here directly shapes retrieval quality — if you minimised squared error on similarities instead, you would not directly maximise the probability of ranking the positive document first.

**Agentic systems (planning, tool use, acting in loops).** Agents that call tools and act in the world are often trained with **reinforcement learning**, where the loss is a *policy gradient* — but the underlying objective is still a loss you choose. For tool-use, the loss is typically a **maximum likelihood objective** (cross-entropy) on the sequence of actions: given the current state (including tool outputs), the model should assign high probability to the correct next action. If the agent is rewarded for completing a task, a common loss is the **negative expected reward** (a scalar to minimise), and the training algorithm (like REINFORCE or PPO) optimises that. The choice of reward function itself is a loss-function choice: if you reward only task completion, the agent may take shortcuts; if you reward progress, it may learn more robustly. A developer writing an agent will implement a loss like `policy_loss = -log_prob(action) * advantage`, where `advantage` is a scalar measuring how much better/worse this action was than expected — this is a cross-entropy-like loss weighted by the advantage.

**Everyday applied machine learning.** In regression (predicting house prices, temperature, demand), the default is **mean squared error (MSE)**, but you should choose based on your business metric. If you care about the *median* rather than the mean — for example, predicting delivery times where a few extreme delays should not dominate — use **mean absolute error (MAE)** or **Huber loss**. If you are predicting probabilities (click-through rate, fraud probability), use **binary cross-entropy** (a special case of cross-entropy), not squared error, because it rewards calibrated probabilities. Your library notes say: 'A mismatch between loss and business metric guarantees optimising the wrong thing.' Concretely: suppose you are a ride-hailing company predicting trip duration; you charge customers based on the estimate, so an overestimate loses you business and an underestimate loses you money. If you minimise MSE, you treat a 5-minute underestimate and a 5-minute overestimate identically. If your business loses ₹100 for every minute over but only ₹10 for every minute under, you should use an **asymmetric loss** that penalises underestimates 10 times more. A developer might implement a custom loss that multiplies the error by a factor depending on its sign. The library goes further: 'Squared error punishes outliers heavily; absolute error and Huber do not.' In fraud detection, a few extreme cases (huge fraudulent transactions) should be caught, so MSE can be appropriate because it forces the model to focus on those outliers. In a recommendation system, the loss is often **binary cross-entropy** for click prediction, or **ranking losses** like pairwise margin loss — these are chosen because they directly optimise the order of recommendations, not the exact predicted value.

**What breaks when done badly.** If you choose a loss that does not match your metric, the model will be excellent at the loss and poor at your task. Example: training a weather-forecast model with MSE when your business only cares whether it will rain (binary classification) — the model will optimise temperature accuracy and ignore rain probabilities, because MSE does not reward correct rain/no-rain discrimination. If you use cross-entropy for a regression task (outputs are not probabilities), the loss may not be defined or may behave nonsensically. If you use a loss that is not differentiable (e.g., raw accuracy is a step function), gradient descent cannot optimise it — you need a smooth proxy. This is why even when the business metric is 'accuracy' or 'F1', the training loss is almost always a smooth surrogate like cross-entropy.

**What a developer actually writes.** In PyTorch, choosing a loss is often one line: `criterion = nn.CrossEntropyLoss()` or `criterion = nn.MSELoss()`, and then inside the training loop `loss = criterion(output, target)`. For a custom loss, you write a small function in Python:

```python
def asymmetric_mae(pred, true):
    error = true - pred  # positive if under-predicting
    return torch.where(error > 0, error * 3, error * 1).abs().mean()
```

This encodes 'under-predictions are three times as costly as over-predictions.' The library notes the key pitfall: the loss must be differentiable (or at least have a subgradient) for backpropagation to work — everything after the loss in the backward pass depends on its gradient.


---

Read this on the web: https://basavapurushottam.com/learn-ai/loss-function-selection
