# Mutual Information

How much knowing one variable reduces uncertainty about another, capturing any dependence rather than only linear correlation.

**Rests on:** Data Compression

**Leads to:** Gini and Information Gain

*Mathematical Foundations · Information Theory*


## Why learn this

You want to know if two things in your professional life are genuinely connected or just coincidentally appearing together. For example, does a customer reading page A actually make them more likely to buy product B? Or is that just noise? Mutual information is the specific metric that answers this: it tells you exactly how much knowing one piece of data reduces your uncertainty about another. It matters now because it allows you to filter signal from noise. If you are trying to decide which sales metrics matter, mutual information shows you which variables actually carry information about your goal (like revenue) and which are just random chatter. It works for any kind of relationship—whether the connection is a straight line or a complex curve—and it gives you a single number that quantifies that strength in terms of saved uncertainty.


## The intuition

Imagine you are trying to guess whether it will rain in Mumbai tomorrow. Without any information, your uncertainty is high; you have no idea. Now I tell you the sky is dark and heavy with clouds. Your uncertainty collapses: it is probably going to rain. Knowing the cloud cover reduced your uncertainty about rain. That reduction in uncertainty is exactly what mutual information measures.

To see this numerically, imagine we track weather data for 100 days. On 50 of those days, it rained (rain happens half the time). Your initial 'uncertainty' is high because there are many possible outcomes. But if I tell you that on a specific day, the cloud cover was above 90%, and we know that in our historical data, it rained on all 20 days with such heavy clouds, your uncertainty drops to near zero. The mutual information between 'cloud cover' and 'rain' is high.

Contrast this with another variable: the day of the week. We look at the same 100 days. Whether it was Tuesday or Wednesday never changed the likelihood of rain; on average, rain happened 50% of the time regardless of the weekday. Here, knowing the day of the week tells you nothing new. The mutual information between 'day of week' and 'rain' is zero.

Crucially, this metric captures *any* pattern of dependence, not just a straight-line relationship. Imagine two variables that form a perfect circle: if you know the X-coordinate of a point on the circle, you know the Y-coordinate must be either positive or negative (two possibilities), drastically narrowing down the range of values. Linear correlation would see no relationship here because there is no slope, but mutual information would catch that strong dependency entirely.


## An analogy

Think of mutual information as a measure of shared insight between two colleagues reporting on a project. Imagine Alex and Sam each write a daily report with a number representing their progress. You are trying to guess Sam's number without seeing his paper. Initially, you have maximum uncertainty—you know nothing.

Now, suppose you are allowed to peek at Alex's number. If knowing Alex's progress allows you to predict Sam's progress with high accuracy (perhaps because they work on the same dependent tasks), then there is a high mutual information between their reports. You have 'saved' yourself from having to guess blindly.

If, however, Alex's numbers are totally random and unrelated to Sam's, peeking at Alex’s report changes nothing about your ability to guess Sam's. The mutual information is zero. The key feature is symmetry: if seeing Alex's number helps you guess Sam's, then seeing Sam's number equally helps you guess Alex's. It is a shared reduction in uncertainty for both parties.

This analogy holds up well for understanding the concept, though note that it focuses on single instances; mutual information actually looks at the aggregate pattern of how often numbers appear together across many days, not just one lucky match.


## Definition

Mutual information is the amount by which knowing the value of one variable reduces your uncertainty about another, measured in units called bits, and it equals zero if and only if the two variables are completely unrelated.


## How it connects

This concept sits beside the idea of entropy. Entropy measures the average uncertainty or 'surprise' in a single variable on its own—for example, how unpredictable the weather is on any given day. Mutual information is essentially the overlap between two such entropies; it tells you how much uncertainty about one variable is resolved by the other. It also connects directly to feature selection in data analysis. When you have a large list of potential predictors for an outcome, mutual information helps you rank them by how much information they actually provide about that outcome, allowing you to discard irrelevant variables effectively.


## The mathematics

Mutual information measures how much knowing one variable reduces your uncertainty about another. It is built from two simpler ideas: entropy, which measures total uncertainty, and conditional entropy, which measures remaining uncertainty after you know something else.

Think of entropy as the number of yes/no questions needed to guess an outcome. A fair coin has 1 bit of entropy because you need one question (Heads or Tails?) to know the result. If a coin always lands Heads, you need zero questions because you already know the answer; its entropy is 0.

Mutual information compares two entropies:
1. The uncertainty of variable Y on its own, written H(Y).
2. The uncertainty of Y after you have looked at variable X, written H(Y|X).

The mutual information I(X;Y) is the difference between these two numbers:
I(X;Y) = H(Y) - H(Y|X)

This difference tells you how many yes/no questions you saved by knowing X. If X and Y are completely unrelated, knowing X changes nothing about your uncertainty of Y. The two entropies are equal, the difference is 0, and the mutual information is 0.

To calculate this precisely, we use a formula that averages the "surprise" of every possible pair of outcomes. The standard formula for mutual information between two variables X and Y is:

I(X;Y) = the sum over all x and y of [ p(x,y) times log base 2 of (p(x,y) divided by (p(x) times p(y)) ) ]

Here is what each part means:
- p(x,y) is the probability that X takes value x AND Y takes value y at the same time.
- p(x) is the probability that X takes value x, ignoring Y.
- p(y) is the probability that Y takes value y, ignoring X.
- The term inside the logarithm, p(x,y) divided by (p(x) times p(y)), compares the actual joint probability to what it would be if X and Y were independent. If they are independent, this ratio is 1, and log(1) is 0, so mutual information is 0.
- We multiply this log value by p(x,y) because outcomes that happen more often contribute more to the average.
- The "sum over all x and y" means we do this calculation for every possible combination of values X and Y can take, and add them all up.


## A worked example

Suppose we have two coins, A and B. We flip them together many times. We want to measure how much knowing the outcome of coin A reduces our uncertainty about coin B.

The possible outcomes for a pair of flips are:
- Both Heads (HH)
- A is Heads, B is Tails (HT)
- A is Tails, B is Heads (TH)
- Both Tails (TT)

The observed probabilities for these four outcomes are:
- P(HH) = 0.4
- P(HT) = 0.1
- P(TH) = 0.1
- P(TT) = 0.4

Notice the pattern: if A is Heads, B is much more likely to be Heads (0.4 vs 0.1). If A is Tails, B is much more likely to be Tails. This dependence means knowing A helps predict B.

Step 1: Calculate the marginal probabilities for A and B.
To find p(A=Heads), add the probabilities of all outcomes where A is Heads:
p(A=Heads) = P(HH) + P(HT) = 0.4 + 0.1 = 0.5
To find p(A=Tails):
p(A=Tails) = P(TH) + P(TT) = 0.1 + 0.4 = 0.5

To find p(B=Heads), add the probabilities of all outcomes where B is Heads:
p(B=Heads) = P(HH) + P(TH) = 0.4 + 0.1 = 0.5
To find p(B=Tails):
p(B=Tails) = P(HT) + P(TT) = 0.1 + 0.4 = 0.5

Step 2: Calculate the entropy of B alone, H(B).
Entropy is the average surprise. For each outcome, calculate -p * log2(p), then sum them.
For B=Heads: probability is 0.5. log2(0.5) = -1. So, -0.5 * (-1) = 0.5.
For B=Tails: probability is 0.5. log2(0.5) = -1. So, -0.5 * (-1) = 0.5.
H(B) = 0.5 + 0.5 = 1 bit.
This means without any extra information, you need 1 yes/no question to guess B.

Step 3: Calculate the conditional entropy H(B|A).
This is the average uncertainty of B after knowing A. We calculate the entropy of B for each case of A, then weight by how likely that case of A is.

Case 1: A is Heads.
We look only at outcomes where A is Heads: HH (prob 0.4) and HT (prob 0.1).
The total probability of A being Heads is 0.5.
The conditional probability that B is Heads given A is Heads is:
p(B=Heads | A=Heads) = P(HH) / p(A=Heads) = 0.4 / 0.5 = 0.8
The conditional probability that B is Tails given A is Heads is:
p(B=Tails | A=Heads) = P(HT) / p(A=Heads) = 0.1 / 0.5 = 0.2

Now calculate the entropy for this specific case, H(B|A=Heads):
For B=Heads: -0.8 * log2(0.8). log2(0.8) is approximately -0.3219. So, -0.8 * (-0.3219) = 0.2575.
For B=Tails: -0.2 * log2(0.2). log2(0.2) is approximately -2.3219. So, -0.2 * (-2.3219) = 0.4644.
H(B|A=Heads) = 0.2575 + 0.4644 = 0.7219 bits.

Case 2: A is Tails.
We look only at outcomes where A is Tails: TH (prob 0.1) and TT (prob 0.4).
The total probability of A being Tails is 0.5.
The conditional probability that B is Heads given A is Tails is:
p(B=Heads | A=Tails) = P(TH) / p(A=Tails) = 0.1 / 0.5 = 0.2
The conditional probability that B is Tails given A is Tails is:
p(B=Tails | A=Tails) = P(TT) / p(A=Tails) = 0.4 / 0.5 = 0.8

Now calculate the entropy for this specific case, H(B|A=Tails):
For B=Heads: -0.2 * log2(0.2) = 0.4644.
For B=Tails: -0.8 * log2(0.8) = 0.2575.
H(B|A=Tails) = 0.4644 + 0.2575 = 0.7219 bits.

Step 4: Average the conditional entropies.
H(B|A) is the weighted average of the entropies from Case 1 and Case 2, using the probabilities of A as weights.
p(A=Heads) = 0.5
p(A=Tails) = 0.5
H(B|A) = (0.5 * H(B|A=Heads)) + (0.5 * H(B|A=Tails))
H(B|A) = (0.5 * 0.7219) + (0.5 * 0.7219)
H(B|A) = 0.36095 + 0.36095 = 0.7219 bits.

Step 5: Calculate Mutual Information I(A;B).
I(A;B) = H(B) - H(B|A)
I(A;B) = 1.0 - 0.7219
I(A;B) = 0.2781 bits.

Result: The mutual information is approximately 0.278 bits. This positive number confirms that A and B are dependent. Knowing the outcome of coin A reduces the uncertainty about coin B by about 0.278 bits.


## From scratch, in Python

```python
import math

# We need a list of joint probabilities. Each entry is a tuple: (x, y, p)
# where x and y are values, and p is the probability of that pair.
# For the coin example:
joint = [('H', 'H', 0.4), ('H', 'T', 0.1), ('T', 'H', 0.1), ('T', 'T', 0.4)]

# First, compute marginal probabilities: P(X=x) and P(Y=y)
# We'll use dictionaries to look up by label.
marginal_x = {}
marginal_y = {}
for x, y, p in joint:
    marginal_x[x] = marginal_x.get(x, 0) + p
    marginal_y[y] = marginal_y.get(y, 0) + p

# Now compute mutual information using the formula:
# I = sum over all pairs of p(x,y) * log2( p(x,y) / (p(x)*p(y)) )
mi = 0.0
for x, y, p in joint:
    mi += p * math.log2(p / (marginal_x[x] * marginal_y[y]))

print("Mutual information:", mi)
# Output: Mutual information: 0.2780719051126378
```


## What the code does

We start with a list of tuples. A tuple is like a list, but it cannot be changed after you make it. Here, each tuple holds the joint outcome and its probability: ('H', 'H', 0.4) means the probability of X=heads and Y=heads is 0.4. The for loop goes through each tuple and adds the probability to the appropriate dictionary — a dictionary is like a lookup table that maps a label to a value. For marginal_x, we are summing probabilities for each value of X, ignoring Y. After the loop, marginal_x['H'] is 0.5 and marginal_x['T'] is 0.5. The second loop computes the sum in the formula: for each joint probability p, we compute p divided by (the product of the two marginals), take the base-2 log, multiply by p, and add. The math.log2 function is the base-2 logarithm, the same one in the formulas. The result is exactly the 0.278 bits we got by hand. This plain-Python version is clear, but for large datasets you would use NumPy to do the same operations on entire arrays at once, much faster.


## Common misconceptions

**Mutual information is the same as correlation** — If I have a correlation of zero, there's no relationship, so mutual information must be zero too.

Correlation only measures linear relationships. Two variables can be perfectly dependent in a nonlinear way — like a circle — and have zero correlation. In fact: Mutual information captures any kind of dependence. Zero correlation does not imply independence; zero mutual information does.

**Mutual information tells you the direction of the relationship** — Mutual information tells me whether X causes Y or Y causes X.

Mutual information is symmetric — I(X;Y) = I(Y;X) — and it measures only the strength of the association, not which variable is the cause. In fact: Mutual information measures how much knowing one helps you guess the other, in both directions equally. To infer causation, you need a causal model and assumptions, not just this statistic.

**Mutual information is a distance** — Mutual information measures how far apart two distributions are, so I can use it as a distance metric in clustering.

Mutual information is zero when variables are independent, but it grows without bound as dependence increases — it is not bounded above and is not a metric (it does not satisfy the triangle inequality or symmetry in the usual sense). In fact: Mutual information measures how much *shared information* there is, not how different two things are. To measure distance between distributions, use KL divergence or a proper metric like the Jensen-Shannon divergence.

**Estimating mutual information from data is easy — just plug the numbers in** — I can compute mutual information directly from my continuous data by just calculating the formula with the observed frequencies.

Continuous data needs binning or density estimation, and naive binning introduces bias — the mutual information estimate grows with the number of bins, even if the true value is zero. In fact: Reliable estimation of mutual information from continuous data is a hard statistical problem. Modern methods use k-nearest-neighbour estimators or neural estimators, and they still come with the risk of high variance. For discrete data, the formula works exactly, but for continuous, you need careful preprocessing.

**Mutual information is always the same as the entropy of one variable** — Since mutual information is the reduction in uncertainty, it should be as large as the entropy of the first variable.

Mutual information is bounded above by the smaller of the two entropies. If one variable is a coin flip (1 bit) and the other is a 6-sided die (2.6 bits), the mutual information can be at most 1 bit, because the coin alone cannot reduce the die's uncertainty by more than 1 bit. In fact: The maximum mutual information between X and Y is min(H(X), H(Y)). It is the shared secret, and you cannot share more than you have.


## In short

Mutual information measures how much knowing one variable reduces uncertainty about another, capturing any dependence, linear or not. It is symmetric, non-negative, and zero exactly when the variables are independent. It is computed as the difference between entropy and conditional entropy, or equivalently as the average log-ratio of the joint probability to the product of the marginals. It is used in feature selection, compression, and as a fundamental concept behind training objectives like cross-entropy.


## Where this is used

Mutual information has a specific and powerful role in building AI systems. 

**Feature selection in applied machine learning** — When a developer builds a model to predict whether a customer will buy, or whether a transaction is fraud, they have many possible input features: age, income, time of day, location, purchase history. A common problem is that many features are irrelevant or redundant, and feeding them all in can confuse the model and slow it down. The developer computes the mutual information between each candidate feature and the target variable, ranks the features by that value, and keeps only the top k. This works even when the relationship is not linear — for example, a fraud pattern that only appears when two features are in a certain range. A developer writes something like `mutual_info_classif(X, y)` from scikit-learn, which estimates the mutual information between each column of X and the target y, and then picks the features with the highest scores. What breaks when it is done badly: estimating mutual information from continuous data is hard and noisy, so a naive estimate can rank random noise above a real signal, throwing away the useful feature. That is why the library versions use tricks like binning the data into buckets and applying a correction for the number of buckets. 

**Large language models** — Mutual information is not directly used in training GPT-style models, but the same family of ideas underpins the attention mechanism. In attention, the model learns to weigh how much one word should influence another — that is, how much knowing one token reduces uncertainty about another in the current context. The attention weights are learned, but the principle is exactly mutual information: how much does this word tell me about the next word? The cross-entropy loss used in training is also intimately related to mutual information: minimising cross-entropy between the predicted distribution and the true distribution is equivalent to maximising the mutual information between the input sequence and the output token. So while a developer does not call a function called 'mutual_information' inside an LLM, the objective they optimise is built from the same foundation. 

**Small models on devices** — On a phone or a Raspberry Pi, memory and battery are tight. A developer who wants to compress a model might use mutual information to prune away weights or neurons that carry little information about the output. This is a hot research area: train a model to keep only the information that matters, maximising mutual information between the compressed representation and the target while minimising it between the representation and the input (the so-called information bottleneck). In practice, a developer might use a library like `torch.nn` and write custom regularisation terms that penalise the information flow. What breaks: if you prune too aggressively, the model loses the ability to generalise because you have thrown away information that was not directly about the label but helped in unseen situations. 

**Retrieval-augmented generation (RAG)** — RAG systems retrieve relevant passages from a database and feed them to a language model to answer a query. The retrieval step is often done by embedding both the query and the passages into vectors and computing similarity — usually cosine similarity, which is closely related to the dot product. But mutual information offers a more principled way to rank. A passage that is highly correlated with the query in a linear way might still be irrelevant; mutual information would capture nonlinear dependencies. In practice, libraries like `sentence-transformers` implement embedding similarity, and some research uses mutual information to re-rank. A developer might write a custom scorer that estimates mutual information between the query and each passage using a histogram of co-occurrence counts, and then uses that score to order results. The catch: estimating mutual information between two long texts is computationally heavy and noisy, so most systems still use cosine similarity, and mutual information is used offline to evaluate or tune the embedding model. 

**Agentic systems that plan and call tools** — When an agent decides which tool to call (e.g., whether to search the web, run a calculator, or read a file), it is effectively asking: which action will reduce my uncertainty about the best response? This is exactly the decision-theoretic use of mutual information. In systems that use Bayesian optimisation or active learning, the agent picks the query that maximises the expected reduction in entropy — that is, the query with the highest expected mutual information. For example, an agent exploring a new API might first call the function that gives the most information about the API's behaviour. A developer might implement this by maintaining a probability distribution over possible outcomes and, for each candidate action, computing the mutual information between the current belief and the outcome after taking that action, then choosing the action with the highest value. Libraries like `bayesian-optimization` or custom probabilistic models in PyTorch are used. When it breaks: if the mutual information estimate is wrong, the agent wastes time calling uninformative tools, and the whole system slows down or fails. 

**Data compression and communication** — This is the original application. In any system that transmits data — a video call, a weather satellite, a streaming service — mutual information tells you how much redundancy exists between two variables (e.g., between pixel values in a video frame). Lossless compression works by removing redundancy; the theoretical limit is the entropy. If you want to compress two correlated streams together, mutual information tells you how many bits you can save by encoding them jointly rather than separately. Developers of codecs like H.264 or JPEG use these ideas all the time. In practice, a developer writing a compression library might use a function from `zlib` or a custom arithmetic coder, but the design decisions about how to model dependencies come straight from information theory. 

**Day-to-day debugging** — A developer might use mutual information to check whether two variables in their pipeline are actually independent, which is useful when building causal models or testing data generators. For instance, if you have a simulator that generates synthetic data for training, you can verify that it produces the correlations you expect by computing mutual information between the generated features. A simple implementation might use `sklearn.feature_selection.mutual_info_regression` to score each feature against the target and confirm it matches your design. 

Where mutual information genuinely does not appear: it is not used in the core forward pass of neural networks or in backpropagation. It plays a supporting role in training objectives, evaluation, and feature engineering, not in the arithmetic of a forward pass. A false claim would be to say that every neural network layer computes mutual information — it does not. The connection is more subtle: the training objective is derivative of it, and the models implicitly learn to preserve mutual information in their representations.


---

Read this on the web: https://basavapurushottam.com/learn-ai/mutual-information
