# Shannon Entropy

The average surprise in a distribution, measured in bits, and the theoretical floor on lossless compression of samples from it.

**Leads to:** Cross-Entropy Loss, Data Compression, ELBO

**Rests on nothing.** A place to start.

*Mathematical Foundations · Information Theory*


## Why learn this

Imagine you want to send a message to a friend, or save a photo on your phone. Every piece of information takes up space — bits in a file, or time on a call. Entropy tells you the absolute minimum amount of space needed to store or transmit that information without losing anything. If you understand entropy, you understand why some data is easy to shrink and other data resists compression. It answers the question: what is the most efficient way to write down what I just saw? For example, if you are sending a long string of letters, and some letters appear very often (like 'e') while others rarely do, entropy helps you figure out how many bits each letter needs in an optimal system. If you don't account for this efficiency, you waste space and time.


## The intuition

Think of a coin. If it is fair — heads and tails equally likely — then every toss genuinely surprises you: you had no way to guess better than a coin flip. If the coin is so biased that it lands heads 99 times out of 100, then a head is hardly a surprise at all; a tail, on the other hand, is a big surprise. Entropy is a way of putting a number on 'how surprising, on average, are the outcomes in this situation?' The more evenly spread the probabilities, the higher the surprise on average, and the higher the entropy. If one outcome is near-certain, the average surprise is low — you almost always know what is coming. That is the whole idea, before any maths: entropy measures how uncertain you are, on average, before you see the next outcome. Uniform distributions (all outcomes equally likely) have the maximum entropy; concentrated distributions (one outcome dominates) have low entropy.


## An analogy

Imagine you are playing a guessing game with a friend who silently picks one of four colours: red, blue, green, yellow. If all four are equally likely, you have no clue which one it is — you need to ask 'is it this one?' questions, and on average it takes 2 questions to pin it down. That 2 is the entropy, in bits. Now suppose your friend is a fan of red and picks red 90% of the time. Most of the time you can guess red and be right; you rarely need many questions. The average number of questions drops, and so does the entropy. In fact, if red is 90%, blue 5%, green 3%, yellow 2%, the best strategy — ask 'is it red?' first — gives an average of about 0.6 questions. That 0.6 is the entropy in bits. This analogy works beautifully for a while: entropy really is the average number of yes/no questions needed to identify an outcome, and it is exactly the number of bits needed to store the outcome in an optimal code. But it breaks down if you push it too far. Real codes work on long streams of symbols, not one symbol at a time, and the 'questions' in the analogy assume you can design the best possible questioning strategy. Also, entropy can be a fractional number of bits (like 0.6), even though you can only ask a whole number of questions for a single outcome — the fractional bits come from amortising over many outcomes. So treat the guessing game as a good ladder, not the destination.


## Definition

Shannon entropy is the average uncertainty or 'surprise' in a set of possible outcomes. It represents the theoretical minimum average number of bits needed to store any item from that set in a perfectly efficient system.


## How it connects

You are starting from a clean slate — no prior knowledge of probability, functions, or algebra is assumed here. The only branch of mathematics you need now is arithmetic: adding, multiplying, and logarithms (which we will teach from scratch when we need them). Shannon entropy is the first concept in information theory. From this seed, you can understand how data compression works by assigning shorter codes to frequent items, and how prediction errors are measured by comparing what you expect with what actually happens. Your notes record that entropy is 'the average surprise in a distribution' — that is exactly the intuition you just built.


## The mathematics

Entropy is a way of measuring uncertainty. It works with a list of probabilities — numbers between 0 and 1 that add up to 1. For each probability, you compute a 'surprise' value: the lower the probability, the higher the surprise. The exact rule is that surprise is the negative of the logarithm (base 2) of the probability. A probability of 1/2 gives a surprise of 1; a probability of 1/4 gives a surprise of 2; a probability of 1/8 gives a surprise of 3. Why? Because if something has probability 1/2 it needs 1 bit to say which of two equally likely outcomes happened; probability 1/4 needs 2 bits; probability 1/8 needs 3 bits. The entropy is then the weighted average of these surprises: you take each probability, multiply it by its own surprise, and add them all up. That number is the average number of bits needed to represent an outcome from this distribution. It is also the floor on lossless compression — no compression scheme can do better on average.

H(X) = -Σᵢ pᵢ log₂(pᵢ) = Σᵢ pᵢ log₂(1/pᵢ)


## A worked example

```
Consider a distribution over four outcomes A, B, C, D with probabilities 0.5, 0.25, 0.125, 0.125 (they add up to 1). We will compute the entropy by hand, in bits.

Step 1: outcome A: 0.5 × (log₂(1/0.5)) = 0.5 × 1 = 0.5   # log₂(1/0.5) = log₂(2) = 1, so the surprise is 1 bit. Weight by probability 0.5.
Step 2: outcome B: 0.25 × (log₂(1/0.25)) = 0.25 × 2 = 0.5   # log₂(4) = 2, so surprise is 2 bits. Weight by 0.25.
Step 3: outcome C: 0.125 × (log₂(1/0.125)) = 0.125 × 3 = 0.375   # log₂(8) = 3, so surprise is 3 bits. Weight by 0.125.
Step 4: outcome D: 0.125 × 3 = 0.375   # Same as C, since D also has probability 0.125.
Step 5: Entropy = 0.5 + 0.5 + 0.375 + 0.375 = 1.75   # Sum of the weighted surprises.

Result: The entropy is 1.75 bits. This means that on average, you need 1.75 bits to represent each outcome from this distribution, and no lossless compression scheme can beat that average.
```


## From scratch, in Python

```python
def shannon_entropy(probabilities):
    total = 0.0
    for p in probabilities:
        if p > 0:
            total += p * log2(1 / p)
    return total

from math import log2

probs = [0.5, 0.25, 0.125, 0.125]
print(shannon_entropy(probs))  # prints 1.75
```


## The same thing, with a framework

There is no dedicated framework for computing Shannon entropy; it is a single mathematical formula. In NumPy, which is a library for working with numbers in bulk, you can compute it in one line:

```python
import numpy as np
probs = np.array([0.5, 0.25, 0.125, 0.125])
H = -np.sum(probs * np.log2(probs))
print(H)  # 1.75
```

Here, `np.log2(probs)` computes the base-2 logarithm of every element in the array at once, `probs * np.log2(probs)` multiplies two arrays element-wise, `np.sum` adds all the values, and the leading negative sign applies the formula −Σp log₂ p. Underneath, NumPy is doing exactly the same loop as our plain Python function — it is just written in C for speed and expressed more compactly.


## What the code does

We define a function called `shannon_entropy` that takes a list of probabilities. A list is exactly what it sounds like: a container that holds items in order, like a shopping list. Here it holds the probabilities. `total = 0.0` starts a running total at zero. The `for p in probabilities:` loop goes through each probability in the list one at a time, with `p` taking on the value of each element in turn. Inside the loop, `if p > 0:` checks that the probability is positive, because log₂(1/p) is undefined for p = 0 (we would be dividing by zero). If it is positive, we add `p * log2(1 / p)` to the running total. `log2` is a function from the `math` library that computes the base-2 logarithm — the same one we used in the numerical example. The line `from math import log2` makes that function available. After the loop, the function returns the total, which is the entropy. We then create a list called `probs` with the four probabilities, and call the function on it. The `print(...)` statement shows the result. This is a direct translation of the formula: for each probability, multiply it by its own surprise, and add them all up.


## Common misconceptions

**Entropy means disorder or chaos.** — Entropy is just another word for messiness, like a messy room.

In physics, entropy is related to disorder, but in information theory it is a precise measure of uncertainty or surprise. A messy room has no associated probability distribution, but a fair coin has exactly 1 bit of entropy. Confusing the two leads to misunderstanding what the number measures. In fact: Entropy measures the average surprise of outcomes from a probability distribution. It is a number, in bits, that tells you how uncertain you are before you observe the next sample.

**High entropy means more information is always better.** — High entropy must be good because it means the data is rich.

High entropy means each sample is genuinely surprising — you learn a lot from each one, but it also means the data is hard to compress and hard to predict. For many tasks, low entropy (predictable) is what you want, like a model that is confident about its predictions. In fact: Entropy is not a value judgment. It is a description of how unpredictable a distribution is. Its usefulness depends on the goal: for compression you want low entropy, for diversity you might want high entropy.

**Entropy is the same as the number of possible outcomes.** — If there are 16 possible outcomes, the entropy must be 16.

Entropy depends on the probabilities, not just the count. Even with 16 possible outcomes, if one has probability 0.9 and all others share the rest, the entropy is much less than log₂(16)=4 bits. In fact: Entropy is measured in bits, and it is at most the logarithm (base 2) of the number of outcomes. It only reaches that maximum when all outcomes are equally likely.

**Entropy requires large amounts of data to calculate.** — I need thousands of samples before I can compute entropy.

The formula for entropy uses probabilities, not samples. You can compute entropy from a known distribution at any time. Estimating entropy from a finite sample is a different problem, but the theoretical definition is a single sum. In fact: Entropy is a property of the probability distribution; it can be computed exactly from the probabilities. Estimating it empirically from data is a separate statistical problem.

**Lossless compression can always be made better with smarter software.** — With a clever enough algorithm, I can compress any file to almost nothing.

Shannon's source coding theorem states that no lossless compression scheme can achieve an average code length shorter than the entropy of the source. This is a mathematical floor, not a technological limitation. In fact: Entropy is the fundamental limit for lossless compression. Any scheme can approach it but never beat it, because doing so would violate the conservation of information.


## In short

Shannon entropy is the average surprise (in bits) of a distribution, computed as the sum of p×log₂(1/p). It is the theoretical lower bound on lossless compression. Uniform distributions have maximum entropy; concentrated distributions have low entropy. It is the foundation of cross-entropy loss, KL divergence, and mutual information, and it appears in everything from language model training to agent uncertainty. Remember the formula, remember the guessing-game intuition, and remember that it is a floor, not a suggestion.


## Where this is used

Shannon entropy appears everywhere in AI, though often under the surface. Here is how it is used in practice, and what a developer actually does.

**Large language models (LLMs).** The training objective for an LLM is cross-entropy loss, which is directly built on entropy. When the model predicts the next token, it outputs a probability distribution over the entire vocabulary. The loss compares this predicted distribution to the true one-hot distribution (which has probability 1 on the actual next token, 0 elsewhere). Cross-entropy loss is literally the entropy of the true distribution plus the KL divergence between the predicted and true distributions. Minimising cross-entropy is equivalent to minimising the KL divergence. In practice, a developer uses a framework like PyTorch and calls `torch.nn.CrossEntropyLoss()`, passing the model's logits and the target token indices. The framework computes the loss internally; the developer rarely writes the formula, but understanding it tells you why the loss penalises a confident wrong prediction far more than an uncertain one — because the log of a tiny probability is a huge negative number.

**Small models running on phones or laptops.** When a model must run on-device, its size matters — every weight takes storage and memory. Entropy gives a hard floor on how small the model's weights can be stored losslessly. A developer might quantise weights (reduce the precision from 32-bit floats to 8-bit integers) which is lossy, but the entropy of the weight distribution tells you how many bits per weight are truly necessary if you want lossless compression. Frameworks like TensorFlow Lite or PyTorch Mobile use entropy-aware compression to shrink models; understanding entropy helps you read their documentation and know why some compression ratios are possible and some are not.

**Retrieval-augmented generation (RAG).** RAG systems retrieve relevant documents and feed them to an LLM. The retrieval step often relies on embeddings and similarity scores, but the decision of how many documents to retrieve can be framed in terms of information gain — which is a reduction in entropy. If retrieving a document reduces the entropy of the model's output distribution a lot, it is valuable. Some advanced RAG systems use entropy-based active learning to decide what to retrieve next. In practice, a developer might use a library like LangChain, which hides this, but the underlying metric (mutual information, derived from entropy) is what the retrieval is optimising.

**Agentic systems that plan and call tools.** An agent that decides which tool to call next often maintains a belief state — a probability distribution over possible states or intents. Entropy of this belief state tells the agent how uncertain it is. If entropy is high, it might choose a clarifying question; if low, it proceeds. This is a direct application of entropy as a measure of uncertainty. In code, a developer might compute entropy over the agent's action probabilities to decide whether to ask the user for clarification. Libraries like LangChain or custom agent loops do this manually.

**Everyday applied machine learning — classification, fraud detection, recommendation, forecasting, speech.** Any classification model is trained with cross-entropy loss, which is entropy plus divergence. That includes fraud detection (classifying transactions), image recognition, speech recognition. In speech recognition, entropy is used in decoding to prune unlikely hypotheses early — if the beam of probabilities has low entropy, the recogniser is confident and can stop early. In recommendation systems, entropy of the distribution over items can be used as a diversity metric — a recommender that always suggests the same few items has low entropy, which might be seen as over-personalisation. A developer using scikit-learn calls `log_loss` (which computes cross-entropy) or writes custom metrics with `scipy.stats.entropy`.

**What breaks when done badly?** Ignoring entropy leads to choosing the wrong loss function (e.g., mean squared error on probabilities instead of cross-entropy), which gives worse training. Overfitting to a low-entropy distribution (like a near-deterministic dataset) can cause a model to be overconfident and miscalibrated. Not understanding the entropy floor means promising lossless compression ratios that are impossible, or underestimating the storage needed for a model.


---

Read this on the web: https://basavapurushottam.com/learn-ai/shannon-entropy
