# Softmax and Logits

Logits are the raw unnormalised scores a network produces; softmax exponentiates and normalises them into a probability distribution.

**Rests on:** Computational Graphs, Forward Propagation

**Leads to:** LLM Nondeterminism

*Deep Learning · Deep Learning*


## Why learn this

You will not get far in AI without understanding how models make choices. Every classifier — whether it decides if an email is spam, which of a thousand images is in front of a camera, or which word follows 'I love you' in a chatbot — ends by turning a jumble of raw numbers into a clean 'this one, with this much confidence'. Understanding this final step means you can read any model's last layer, debug why it is unsure, and later grasp larger ideas like knowledge distillation (inventing a smarter student model) and temperature sampling (making a chatbot more creative). It unlocks **Cross-Entropy Loss**, **Temperature and Sampling**, **Classifier Architectures**, and ultimately **LLM generation mechanics**.


## The intuition

Imagine you have a bowl of three fruits: an apple, a banana, and a cherry. You close your eyes, pick one, and eat it. You cannot explain why, but you have a gut feeling: 'apple 60%, banana 30%, cherry 10%'. A neural network does the same thing, but it starts with rough scores, not percentages. For example, it might give scores like apple: 2, banana: 1, cherry: 0. These raw scores are called **logits** — a technical term for these initial numbers. They are not probabilities because they can be any positive or negative number, and they do not necessarily add up to 1. The softmax function is the recipe that takes these logits and turns them into percentages that add up to 100%. It does two things: it makes sure every percentage is positive and between 0 and 1, and it makes the biggest raw score the biggest percentage. But it does not just take the largest and ignore the rest; it keeps the relative differences. If the apple score is much higher than the banana, the apple percentage will be close to 100%. If they are close, the percentages will be closer. The key trick is that softmax uses the mathematical constant *e* (about 2.718) to amplify differences — a small difference in logits becomes a larger difference in probabilities by raising *e* to the power of each score. That is why it is called 'soft' max — it is a gentle, smooth version of 'pick the largest', not a hard cut.


## An analogy

Think of a talent show. Five contestants perform, and the judges give scores on a scale of 1 to 10. The scores might be: A: 7, B: 8, C: 6, D: 9, E: 5. But the show wants to present the results as percentages of the audience's vote. They cannot just divide each score by the total (7+8+6+9+5=35), because then A would get 20%, B 23%, C 17%, D 26%, E 14%. That works, but it treats a 7 and an 8 as nearly equal. The show's producer, however, wants to reward excellence: they want the winner to stand out more. They introduce a rule: before adding up, they turn each score into something like 'e to the power of the score'. So 7 becomes *e*^7 (a huge number), 8 becomes *e*^8 (even huger), and so on. The differences get amplified: *e*^9 is about 8103, *e*^8 is about 2981, *e*^7 is about 1097, so the ratio between D and A is now 8103/1097 ≈ 7.4, whereas before it was 9/7 ≈ 1.3. The percentages become: D: 73%, B: 27%, A: 10%, C: 3%, E: 0.2% (roughly, after dividing by the total sum of these amplified numbers). This is exactly what softmax does — it raises *e* to the power of each score, then divides each result by the sum of all such results, so the biggest score dominates. This analogy has one limitation: the talent show judges give scores with a limited range, but a neural network's logits can be any number, even negative. Negative logits become very small probabilities after raising *e* to that power, which is fine — it just means 'very unlikely'.


## Definition

Softmax is a function that takes a list of raw scores (called logits) and turns them into a list of probabilities that all sum to 1, by raising *e* to the power of each score and then dividing each result by the sum of all those raised values.


## How it connects

You have not built up much yet, but this connects directly to **Activation Functions** you will learn next — softmax is often called the final activation. It also sets the stage for **Cross-Entropy Loss**, which is the standard way to measure how wrong the probabilities are, and later for **Temperature** — a way to make the probabilities more or less confident. In the learner's library, you have notes on 'Activation Functions' and 'Cross-Entropy Loss' as neighbours — this page is the bridge between them.


## The mathematics

The mathematics here comes from **probability** and **arithmetic**. You need to know how to add, multiply, divide, and use the number *e* (approximately 2.718). You do not need calculus or advanced algebra.

The goal is to turn a list of raw scores (called **logits**) into a list of probabilities. A probability tells us how likely something is, on a scale from 0 to 1. All the probabilities in the list must add up to exactly 1.

The process has three clear steps:

**Step 1: Exponentiate each logit.**
Take each raw score and raise *e* to that power. This is written as *e* raised to the power of *z*. In plain words, we calculate *e^z*. This step makes all numbers positive and stretches out the differences between them. A larger logit becomes a much larger number.

**Step 2: Sum the results.**
Add together all the numbers you got in Step 1. Let us call this total sum **S**. This sum acts as a normalizer, ensuring the final answers fit into the 0-to-1 range.

**Step 3: Divide each exponentiated value by the sum.**
For each class, take its exponentiated value from Step 1 and divide it by the total sum **S**. The result is the probability for that class.

Mathematically, if we have a list of logits *z* with *n* items, written as *z₁, z₂, ..., zₙ*, the probability for the *i*-th class (let us call it *pᵢ*) is:

pᵢ = e raised to the power of zᵢ, divided by the sum of (e raised to the power of zⱼ) for all j from 1 to n.

In symbols often used in textbooks, this looks like:
p(i) = exp(zᵢ) / Σⱼ exp(zⱼ)

Here is what each symbol means:
- *p(i)*: The probability that the input belongs to class *i*. Read as "p of i".
- *zᵢ*: The raw score (logit) for class *i*.
- *exp(zᵢ)*: The exponential function applied to *zᵢ*. This means *e* raised to the power of *zᵢ*.
- *Σⱼ*: The Greek letter Sigma. It means "sum over all j". Here, *j* is just a placeholder index that runs through every class in the list (1, 2, ..., n). So *Σⱼ exp(zⱼ)* means "add up the exponentials of every logit in the list".

**Why does this work?**
Because we divide every individual exponential value by the same total sum **S**, the resulting probabilities are scaled down so they fit between 0 and 1. And because we divided each part by the whole, the parts must add up to the whole (1).

**A useful property: Shifting logits.**
If you add the same number (say, 100) to every logit before starting, the final probabilities do not change. Why? Adding a constant *c* to every logit multiplies every exponential term by *e^c*. When you sum them, the total **S** also gets multiplied by *e^c*. When you divide an individual term (also multiplied by *e^c*) by the total (also multiplied by *e^c*), the *e^c* cancels out. This means softmax only cares about the *relative* differences between logits, not their absolute size.


## A worked example

Let us walk through a concrete example with real numbers.

**The Setup:**
Imagine a neural network is trying to classify an image as one of three things: **Cat**, **Dog**, or **Bird**. The network outputs three raw scores (logits) for these classes:
- Logit for Cat (*z₁*) = 2.0
- Logit for Dog (*z₂*) = 1.0
- Logit for Bird (*z₃*) = 0.0

We want to convert [2.0, 1.0, 0.0] into probabilities.

**Step 1: Exponentiate each logit.**
We use the constant *e* ≈ 2.71828.
- For Cat: Calculate *e* raised to the power of 2.0.
  *e^2.0* ≈ 7.389
- For Dog: Calculate *e* raised to the power of 1.0.
  *e^1.0* ≈ 2.718
- For Bird: Calculate *e* raised to the power of 0.0.
  Any number to the power of 0 is 1. So, *e^0.0* = 1.0

Now we have three new numbers: [7.389, 2.718, 1.0].

**Step 2: Sum the results.**
Add these three numbers together to get the total sum **S**.
**S** = 7.389 + 2.718 + 1.0
**S** = 11.107

**Step 3: Divide each exponentiated value by the sum.**
Now we find the probability for each class by dividing its specific exponential value by **S** (11.107).

- Probability of Cat (*p₁*):
  *p₁* = 7.389 / 11.107
  *p₁* ≈ 0.6652

- Probability of Dog (*p₂*):
  *p₂* = 2.718 / 11.107
  *p₂* ≈ 0.2447

- Probability of Bird (*p₃*):
  *p₃* = 1.0 / 11.107
  *p₃* ≈ 0.0900

**Check:**
Do these probabilities add up to 1?
0.6652 + 0.2447 + 0.0900 = 0.9999 (which is 1.0, allowing for small rounding errors).

**Result:**
The softmax output is approximately [0.665, 0.245, 0.090].
This means the model thinks there is a 66.5% chance it is a Cat, 24.5% chance it is a Dog, and 9.0% chance it is a Bird.

**Observation:**
Notice how the raw scores [2.0, 1.0, 0.0] had differences of 1.0 between them. After softmax, the probability for Cat (0.665) is much higher than for Dog (0.245). The exponential function amplified the difference. The highest logit became the dominant probability.


## From scratch, in Python

```python
import math

# Our logits, one per class. Think of them as the raw scores from a network.
logits = [2.0, 1.0, 0.0]

# Step 1: Exponentiate each logit using the math.exp function (which computes e^x)
exponentials = []
for z in logits:
    exponentials.append(math.exp(z))
    
# Step 2: Sum all the exponentials
sum_exp = sum(exponentials)

# Step 3: Divide each exponential by the sum to get probabilities
probabilities = []
for value in exponentials:
    probabilities.append(value / sum_exp)

print(probabilities)  # Output: [0.6652409557749322, 0.24472847105479767, 0.09003057317027045]

# A more concise way using a list comprehension (a way to build a list in one line):
# exponentials = [math.exp(z) for z in logits]
# sum_exp = sum(exponentials)
# probabilities = [value / sum_exp for value in exponentials]
```


## The same thing, with a framework

```python
import numpy as np

logits = np.array([2.0, 1.0, 0.0])

# numpy's exp works element-wise on arrays, and / broadcasts the division
probabilities = np.exp(logits) / np.sum(np.exp(logits))

print(probabilities)  # [0.66524096 0.24472847 0.09003057]
```


## What the code does

**From scratch:**

First, we import the `math` module, which gives us the `exp` function (raises e to a power). We define `logits` as a list of three numbers. A **list** is a container that holds items in order, and you can change it later — here it holds the three raw scores.

The first `for` loop goes through each item in `logits`, one at a time. For each `z`, we compute `math.exp(z)` — that is e to the power of z — and append it to a new list called `exponentials`. So after the loop, `exponentials` is `[7.38905609893065, 2.718281828459045, 1.0]`.

`sum(exponentials)` adds up all the numbers in the list, giving about 11.107.

The second `for` loop goes through each value in `exponentials`, divides it by `sum_exp`, and appends the result to `probabilities`. That gives us our three probabilities that sum to 1.

**Framework version:**
NumPy is a library for working with lists of numbers (called arrays). `np.array([2.0, 1.0, 0.0])` creates an array. `np.exp(logits)` applies the exponential to every element at once, returning a new array `[7.389, 2.718, 1.0]`. `np.sum(...)` adds all the elements. The division `/` is done element-wise: each number in the first array is divided by the sum. This is exactly the same arithmetic as the plain Python loop, just in fewer lines.


## Common misconceptions

**Softmax is just a fancy way to pick the largest logit** — Softmax only matters for choosing the biggest score; it just tells you which class wins.

Softmax does not just pick the winner — it produces a full probability distribution over *all* classes, preserving the relative differences between all scores. The probabilities are used for sampling, uncertainty estimation, and training loss, not just for argmax. In fact: Softmax converts a vector of raw scores into a vector of probabilities that sum to 1, reflecting the *relative* likelihood of each class. Even the losing classes get a non-zero probability, which is essential for training and for tasks like text generation.

**Logits are already probabilities** — The numbers the network outputs before softmax are already probabilities, like 0.8, 0.1, 0.1.

Logits are raw, unbounded scores — they can be negative, larger than 1, and do not sum to 1. They are not probabilities until transformed by softmax. In fact: Logits are the network's raw confidence scores, like a rating from -infinity to +infinity. Softmax turns them into a valid probability distribution (non-negative, summing to 1).

**Softmax output is the model's true confidence** — If softmax says 0.99, the model is 99% certain and will be right almost all the time.

Softmax probabilities reflect the model's internal *relative* belief, not the true probability of being correct in the real world. Neural networks are often overconfident, especially on out-of-distribution data. In fact: Softmax probabilities are a measure of relative confidence among the classes, but they can be miscalibrated. Calibration techniques (like temperature scaling) are used to align them with actual accuracy.

**Softmax requires all logits to be positive** — Since probabilities can't be negative, logits must be positive before softmax.

Logits can be any real number, including negative. The exponential function maps negative numbers to small positive values, so they still contribute to the distribution. In fact: Logits can be negative, and the exponential function handles that automatically by making them small positive probabilities. The sum of all probabilities is still 1.

**Softmax is the only way to turn scores into probabilities** — The softmax function is the only possible way to convert scores to probabilities; there is no alternative.

There are other functions like sigmoid (for binary classification), or simple normalization (dividing by sum). Softmax is preferred because of its differentiability and the gradient simplification with cross-entropy. In fact: Softmax is one of several ways, but it is standard for multi-class problems because it produces a smooth, differentiable distribution that works well with gradient-based training.


## In short

Logits are raw scores — unbounded, possibly negative, not probabilities. Softmax exponentiates each logit, sums them, and divides each by that sum to get a probability distribution that sums to 1. The magic of the exponential makes larger logits amplify, so the model expresses relative confidence. Only the differences between logits matter — adding a constant changes nothing. This is the final layer of almost every classifier and the engine of next-word prediction in LLMs. Understanding softmax is the key to reading model outputs, controlling creativity with temperature, and later to cross-entropy loss.


## Where this is used

Where do softmax and logits show up in the real world of building AI? Everywhere a model has to choose between a fixed set of options. Here is the detailed map.

**Large language models (LLMs).** When an LLM like ChatGPT generates the next word, it does not simply pick the most likely word. The final layer of the network produces a logit for every word in its vocabulary (often tens of thousands of words). Softmax converts those logits into probabilities. Then the model *samples* from that probability distribution — it picks a word at random, but with higher probability for words with higher softmax scores. This is why the same prompt can give different answers each time. A feature called **temperature** divides the logits by a number before softmax: a low temperature (e.g. 0.2) makes the distribution very sharp (almost always the top word), high temperature (e.g. 1.5) makes it flatter (more randomness). Developers writing with an API like OpenAI's set `temperature` as a parameter — it directly controls the softmax's input. In practice, a developer might write `response = client.chat.completions.create(..., temperature=0.7)` and that single number changes the entire feel of the response.

**Small models running on your phone.** Edge models like on-device keyboards (Gboard's next-word prediction) use the same softmax but with a smaller vocabulary. Because they run on limited hardware, the softmax over many thousands of categories can be a bottleneck. Developers optimize it using a technique called **hierarchical softmax** — instead of computing a probability for every word, they arrange words in a tree and do a series of smaller softmaxes, each over a few options, speeding up prediction. This is an engineering trick, but the core softmax is still there.

**Retrieval-augmented generation (RAG).** When you build a system that retrieves documents from a database to help an LLM answer, the retrieval step often uses a different kind of score — a similarity score between your question and each document. These scores are not probabilities either; they are logits in a loose sense. Some RAG pipelines apply softmax to these similarity scores to get a probability that each document is relevant, which then guides how much weight each retrieved document gets in the final prompt. So a developer might compute `scores = model.encode(questions) @ model.encode(docs).T` (dot products), then `probs = np.exp(scores/τ) / np.sum(np.exp(scores/τ), axis=1)` to get a probability distribution over documents. The temperature τ here controls whether to focus on the top hit or spread attention.

**Agentic systems.** An agent that decides which tool to call (search, calculator, calendar) has a fixed list of possible actions. The policy network produces a logit for each tool, softmax turns it into a probability, and the agent samples or takes the argmax. In a loop, the agent might use temperature to balance exploration (trying new tools) vs exploitation (using what works). If a developer sets `temperature` too high, the agent might call random tools; too low, it might miss creative solutions.

**Everyday applied machine learning.** Classification problems — spam detection, image recognition (is this a cat or dog?), medical diagnosis (which disease?), fraud detection — all end in softmax. In scikit-learn, for instance, a logistic regression's `predict_proba` method uses softmax internally to return probabilities. A data scientist might inspect `model.predict_proba(X)[:, 1]` to get the probability of the positive class, and then set a custom threshold (like 0.7 instead of 0.5) to trade off false positives and false negatives. The softmax itself is so embedded that developers rarely write it by hand — they call `model.predict_proba()` or `torch.softmax(logits, dim=-1)` — but understanding it is essential for debugging why the model is overconfident or underconfident.

**What breaks when done badly:** If you forget to apply softmax and treat raw logits as probabilities, you will get numbers that can be negative or sum to more than 1 — nonsense. If you apply softmax to logits that have not been properly scaled (for instance, if the model is badly calibrated), you get overconfident probabilities: a model might say 99.9% sure when it is wrong half the time. This is why calibration matters — developers use techniques like temperature scaling to fix it. In code, using the wrong `dim` in PyTorch (e.g. softmax over the batch dimension instead of the feature dimension) silently ruins the output, so understanding the shape of your tensors is critical.


---

Read this on the web: https://basavapurushottam.com/learn-ai/softmax-and-logits
