# KL Divergence

The extra cost, in bits, of coding data from one distribution using a code built for another.

**Rests on:** Data Compression

**Leads to:** Cross-Entropy Loss, KL Divergence Asymmetry

*Mathematical Foundations · Information Theory*


## Why learn this

You need to understand KL Divergence because it is the mathematical engine behind how artificial intelligence learns from examples. Imagine you are training a system to recognize handwritten digits. You show it 100 pictures of the number '7' and tell it, "This is what a 7 looks like." The system tries to build its own internal rule for recognizing a 7. KL Divergence is the score that tells you how far off that internal rule is from the reality of what you showed it. If the score is high, the system is confused; if it is low, the system has learned well. This single concept powers large language models like ChatGPT (by helping them predict the next word accurately), recommendation engines (by figuring out which products match your taste), and medical AI (by ensuring diagnoses are precise rather than guessing). It allows you to quantify exactly how much "wrong" a prediction is, measured in bits of information.


## The intuition

Let’s look at a concrete example. Imagine a simple scenario where there are only two possible outcomes for an event: Heads or Tails on a coin flip. A probability distribution is simply a list of the chances for each outcome. For a fair coin, the distribution is [0.5 chance of Heads, 0.5 chance of Tails]. This list must add up to 1 (or 100%). Now imagine an unfair coin that lands on Heads 90% of the time. Its distribution is [0.9 chance of Heads, 0.1 chance of Tails]. These are two different probability distributions because they describe different realities.


## An analogy

Imagine you are a professional chef who knows exactly how to bake a cake using a specific set of ingredients: 3 eggs, 2 cups of flour, and 1 cup of sugar. This is your "true" recipe, or the true distribution. Now, imagine someone else (the model) tries to guess this recipe based on tasting one cake they baked. They come up with a different recipe: 2 eggs, 3 cups of flour, and 1 cup of sugar. This guessed recipe is a different probability distribution because the proportions of ingredients are different.

KL Divergence measures the "waste" or "surprise" you would feel if you followed the wrong recipe. If you have a bag of ingredients perfectly mixed for the true recipe (3 eggs, 2 cups flour) and you try to bake it using the wrong recipe's instructions, you will end up with a bad cake because the proportions don't match. The KL Divergence calculates how much "extra effort" or "surprise" this mismatch causes compared to using the correct recipe.

The analogy works because both cases involve comparing two sets of proportions (ingredients vs. probabilities) and measuring the cost of using one when you should use the other. It fails because, unlike baking where you can count eggs precisely, probability distributions deal with abstract chances that can’t be physically counted, only observed over time.

Note: While this analogy helps visualize the mismatch, remember that KL Divergence specifically measures the average surprise per unit of information, not just a raw count of errors.


## Definition

KL Divergence is the average number of extra bits required to encode data from one probability distribution when using a code optimized for a different probability distribution. In simpler terms, it is the measure of how much one distribution diverges from a second, reference distribution.


## How it connects

This concept sits at the heart of Probability Theory, which is the study of how likely events are to occur, and Information Theory, which is the study of quantifying information. A neighboring topic you will encounter is Cross-Entropy Loss, which is the practical score used by AI models during training; it combines your current uncertainty with the KL Divergence to tell the model how badly it performed on a single example. Another key neighbor is Shannon Entropy, which measures the baseline unpredictability of a system itself, regardless of what code you use.


## The mathematics

The mathematics here comes from a branch called Information Theory, which studies how to measure information and uncertainty. The key idea is that a message's length depends on how surprising it is. A common event carries little information — you already expected it — so it can have a short code. A rare event carries a lot of information, so it needs a longer code. The average length of a code for a distribution is called its entropy — the minimum number of bits you need, on average, to represent one sample from that distribution.

Now suppose you build a code for distribution P — you give short codes to P's common events and long codes to P's rare events. But the data actually comes from distribution Q. The average code length you get is the cross-entropy — the cost of using P's code on Q's data. This cross-entropy is always at least as large as Q's own entropy, because you'd do best with a code designed for Q. The difference — cross-entropy minus Q's entropy — is the KL divergence. It's the extra cost you pay for using the wrong code.

To compute it, you look at each possible event. For each event, you have Q's probability (how often it actually occurs) and P's probability (how likely the code thinks it is). The extra cost for that event is Q's probability multiplied by the log of the ratio Q/P. Sum that over all events. That's the KL divergence.

The logarithm is a mathematical function that turns multiplication into addition, and it measures 'surprise' — a very unlikely event under P has a large positive log, a very likely event has a small or negative log. The log can be in base 2 to get bits, or base e (natural log) for other units — the choice just scales the result.

KL(Q || P) = Σᵢ Q(i) × log₂(Q(i) / P(i))

where:
- Q is the true distribution (the data),
- P is the model distribution (the predictions),
- i runs over all possible events,
- Q(i) is the true probability of event i,
- P(i) is the model's probability of event i,
- log₂ is the logarithm base 2, giving the result in bits.

The double bar || means 'divergence from' — they read it as 'KL from Q to P', not a symmetric distance.


## A worked example

```
We have two simple distributions over two events, A and B. Distribution Q is the true one: A happens 80% of the time, B happens 20%. Distribution P is the model: it thinks A happens 60% of the time, B happens 40%. We want the KL divergence from Q to P — the extra cost of using P's code when Q is true.

Step 1: For event A: Q(A) = 0.8, P(A) = 0.6. Ratio = 0.8/0.6 = 1.333. Log2(1.333) = 0.415. Extra cost = 0.8 × 0.415 = 0.332   # The log of the ratio measures the surprise of seeing A under P. Multiplying by Q(A) weights it by how often A actually happens.
Step 2: For event B: Q(B) = 0.2, P(B) = 0.4. Ratio = 0.2/0.4 = 0.5. Log2(0.5) = -1. Extra cost = 0.2 × (-1) = -0.2   # B is more likely under P than under Q, so the log is negative — the code is shorter than needed for B, but that's offset by the longer code for A.
Step 3: Sum: 0.332 + (-0.2) = 0.132 bits   # The total extra cost is positive, as it must be. The model wastes 0.132 bits on average per sample compared to the true code.

Result: KL divergence from Q to P is 0.132 bits. It is positive because P is not identical to Q. If P matched Q exactly, the log ratios would all be 0 and the sum would be 0.
```


## From scratch, in Python

```python
import math

# Two probabilities for events A and B
q = [0.8, 0.2]   # true distribution
p = [0.6, 0.4]   # model distribution

# Compute KL divergence by hand
kl = 0.0
for i in range(2):
    ratio = q[i] / p[i]
    kl += q[i] * math.log(ratio, 2)   # log base 2

print(kl)   # 0.132
```


## The same thing, with a framework

```python
import numpy as np

q = np.array([0.8, 0.2])
p = np.array([0.6, 0.4])

kl = np.sum(q * np.log2(q / p))

print(kl)  # 0.132

# Or use scipy's built-in function
from scipy.special import rel_entr
kl = np.sum(rel_entr(q, p))
print(kl)  # 0.132
```


## What the code does

The code starts with two lists, `q` and `p`, each holding the probabilities for events A and B. A list is an ordered collection of values — `q[0]` is 0.8, `q[1]` is 0.2. A `for` loop runs over the indices 0 and 1, calculating the ratio `q[i] / p[i]`, then taking the logarithm, multiplying by `q[i]`, and adding to the running total `kl`. The `math.log` function with the second argument 2 does a base-2 logarithm. The loop finishes after two iterations, and the result is printed. This is the direct translation of the formula — no shortcuts.


## Common misconceptions

**KL divergence is a distance metric** — KL divergence measures how different two distributions are, so it must be a distance — it's zero when they're same, and bigger when they're more different.

A true distance metric must be symmetric — the distance from A to B must equal the distance from B to A — and must satisfy the triangle inequality. KL divergence fails both: KL(P||Q) is not the same as KL(Q||P), and the 'distance' can be arbitrarily large in one direction while small in the other. In fact: KL divergence is a measure of relative entropy, not a metric. It is always non-negative and zero only when the distributions are identical, but it is asymmetric. You should always say 'KL from Q to P', not 'KL between Q and P'.

**KL divergence is always finite** — Any two distributions can be compared with KL divergence — it gives a finite number.

If the model assigns zero probability to an event that actually happens (P(i) = 0 while Q(i) > 0), the ratio Q/P is infinite, the log is infinite, and the KL divergence is infinite. This is a real problem in training — a model that never predicts a token that appears in the data will get an infinite loss. In fact: KL divergence is finite only if the model's support (events it gives positive probability) contains the true distribution's support. In practice, developers add a small epsilon to probabilities to avoid infinities.

**Minimising KL divergence is different from maximising likelihood** — There are two separate ways to train a model: one based on likelihood and one based on KL divergence. The choice matters.

Mathematically, maximising the log-likelihood of the data under the model is equivalent to minimising the KL divergence between the empirical data distribution and the model distribution. The two objectives differ by a constant that doesn't depend on the model. In fact: Maximising likelihood and minimising KL divergence are equivalent. This is why cross-entropy loss (which is KL plus the entropy of the data) works as the standard training objective.


## In short

KL divergence measures the extra bits you waste when you use a code built for one distribution to send data from another. It's always non-negative, zero only when distributions match, and asymmetric. Computing it means summing over each event: true probability times log of the ratio of true to model probability. It is the foundation of cross-entropy loss, which is how nearly every AI model is trained, and it appears in distillation, variational inference, and reinforcement learning. Remember: the direction matters, and a zero probability in the model turns the loss into infinity.


## Where this is used

KL divergence is everywhere in modern AI. Here is where, and how.

**Large language models (LLMs).** Every LLM is trained to predict the next token in a sequence. The model outputs a probability distribution over all possible tokens, and the true data has a one-hot distribution — the actual next token has probability 1, all others 0. The loss function is cross-entropy, which is the same as KL divergence from the true distribution to the model's distribution, minus a constant. Minimising cross-entropy minimises KL divergence. The model's weights are adjusted with gradient descent to make its predictions match the data as closely as possible. The result: a model that can generate text that looks like the training data. A developer writing a training loop will see `loss = cross_entropy_loss(logits, targets)` where `logits` are raw scores converted to probabilities, and `targets` are the true token indices.

**Small models on devices (TinyML).** A small model running on a phone or a smart speaker cannot have billions of parameters. One common trick is knowledge distillation: train a small model to mimic a large one. The loss is the KL divergence between the small model's output distribution and the large model's output distribution — the small model is taught to reproduce the large model's probabilities, not just the hard labels. This transfers the 'soft' knowledge (e.g., 'dog' and 'wolf' are more similar than 'dog' and 'toaster') into the small model. A developer uses a framework like TensorFlow Lite with a distillation loss defined as `kl_divergence(teacher_logits, student_logits)`.

**Retrieval-augmented generation (RAG).** When an AI system answers a question by retrieving relevant documents, it needs to rank documents by relevance. Some systems use KL divergence between the query's distribution and a document's distribution over topics to measure similarity. More commonly, the embedding vectors are compared with cosine similarity, but KL divergence appears in the 'reranker' models that re-rank the top retrieved documents — those rerankers are trained with KL divergence losses. A developer writes code that calls a reranker API, but internally it's `kl_divergence(query_context_distribution, document_distribution)`.

**Agentic systems.** An agent that plans and calls tools must decide which tool to use next. It outputs a probability distribution over possible actions. The training loss is often cross-entropy (i.e., KL divergence with a one-hot target). During execution, the agent may also use KL divergence to measure how different its planned behaviour is from a safe or desired behaviour — for example, to penalize deviating from a base policy. In reinforcement learning, the PPO algorithm adds a KL penalty to keep the new policy close to the old one: `loss = -advantage * ratio + β * KL(old_policy, new_policy)`. A developer using a reinforcement learning library sees this as a parameter in the training config.

**Everyday applied machine learning.** In classification, the final layer outputs probabilities, and the loss is cross-entropy — KL divergence again. In topic modelling, Latent Dirichlet Allocation uses a variational objective that includes a KL term. In variational autoencoders (VAEs), the loss is `reconstruction_loss + KL(q(z|x) || p(z))` — the KL term keeps the latent variables close to a standard normal distribution. This is the ELBO decomposition you will learn later. A developer using PyTorch writes a custom loss function that adds a KL term computed via `torch.distributions.kl_divergence`. In fraud detection, models output probabilities of fraud; the loss is cross-entropy, and the model is penalised more for confident wrong predictions on fraud cases — just as the KL divergence gives infinite cost when the model assigns zero probability to something that actually happens.

**What breaks when done badly?** If a model assigns zero probability to an event that occurs in the data, the log ratio goes to infinity — the loss explodes. In practice, developers clip probabilities to avoid division by zero. Also, because KL divergence is asymmetric, choosing the wrong direction changes the model's behaviour: minimising KL(true || model) makes the model cover all modes of the data (mode-covering), while minimising KL(model || true) makes it focus on one mode (mode-seeking). This is why variational inference can miss modes — it minimises reverse KL.


---

Read this on the web: https://basavapurushottam.com/learn-ai/kl-divergence
