# Information Geometry

Treating families of probability distributions as curved manifolds, with divergences playing the role of distance and Fisher information as the metric.

**Rests on:** Fisher Information, Tensors and Shapes

*Mathematical Foundations · Bayesian Methods*


## Why learn this

This topic changes how you visualize probability distributions from flat lists of numbers into shapes in a curved space. Understanding this reveals why some optimization problems are easy while others are treacherously difficult. It explains the geometry behind a method for approximating complex beliefs, called variational inference, and clarifies why gradient-based learning sometimes fails or requires special adjustments when parameters are correlated. Instead of treating your data as points on a graph, you treat them as locations on a landscape where 'distance' means 'how much information do I lose if I confuse these two states?'. This perspective is essential for building robust models in Bayesian Methods that respect the inherent structure of uncertainty.


## The intuition

Imagine you are mapping a mountain range, but your map only gives you coordinates (latitude and longitude), not elevation or slope. You want to know how hard it is to walk from point A to point B on this terrain. On a flat plain, distance is simple: a straight line. But mountains are curved. A step that looks small on your coordinate map might take you up a steep cliff, while another step of the same map-length might slide gently down a slope. The 'true' difficulty of moving between two points depends not just on the coordinates, but on how steep and curved the ground is at that specific location.

Information geometry treats the space of all possible probability distributions exactly like this mountain range. Each distribution—say, a bell curve defined by a mean and a variance—is a point on this landscape. The 'distance' between two distributions is not the difference in their means or variances (the map coordinates). Instead, it is measured by how much information you lose if you pretend one distribution is the other. This measure is called Kullback-Leibler divergence. Think of Kullback-Leibler divergence as a penalty score for being wrong: it quantifies the surprise or error introduced when you use one story to describe reality that follows another.

The shape of this landscape is defined by something called Fisher Information. Think of Fisher Information as a sensor that measures how 'steep' the terrain is at every point. If Fisher Information is high, the landscape is sharply curved; a tiny change in your distribution parameters leads to a large change in the actual probability content (a big jump in the mountain). If Fisher Information is low, the landscape is flat; you can tweak your parameters freely without changing the underlying story much.

For example, consider two bell curves with the same mean but different variances (widths). One is narrow and tall, the other wide and flat. In parameter space, they differ only by one number. But in information geometry, the 'distance' between them depends on how distinct their shapes are. If the curves overlap heavily, the distance is small—they tell similar stories. If they are disjoint, the distance is large. Information geometry gives you the ruler that measures this informational distance, not just the difference in numbers.


## An analogy

Think of a hiker trying to climb a mountain using only a standard flat map with latitude and longitude grid lines. The hiker wants to reach the summit. If they walk one mile north on the flat map, they might find themselves on a gentle meadow in one spot, but face a sheer cliff in another. The 'distance' on the map is misleading because it ignores the terrain's curvature. To climb efficiently, the hiker needs more than the map; they need to know how the slope changes beneath their feet at every step. They need to know not just which direction points up, but how 'steep' that upward path is relative to the ground itself. Information geometry provides this missing context: it scales the steps based on the local curvature (Fisher Information). It tells the hiker: 'Your next step is in a rocky area; even though your map says it's one mile, it will feel like ten miles of effort, so take smaller, more careful steps.' This adjusted direction and step size is what allows optimization algorithms to navigate efficiently without getting stuck on cliffs or wasting time on flat ridges.


## Definition

Information geometry is the study of probability distributions as points on a curved surface where local distances are defined by how much information changes between nearby distributions, rather than by simple differences in numerical parameters.


## How it connects

This concept extends your understanding of Fisher Information by interpreting it not just as a measure of statistical precision, but as the fundamental ruler (metric) that defines distance on the curved space of distributions. It connects directly to Bayesian Inference, where comparing prior and posterior beliefs—or approximate posteriors—requires measuring how 'far apart' these distributions are in terms of information loss, often using Kullback-Leibler divergence.


## The mathematics

Information geometry treats the set of all possible probability distributions as if it were a curved surface, or manifold. Just as you need a map to measure distances on the Earth's curved surface, you need a way to measure 'distance' between two probability distributions.

In flat space, distance is simple. If you move dx units right and dy units up, the total distance is the square root of (dx squared plus dy squared). On a curved surface, this rule changes depending on where you are. You need a local rule, called a metric, that tells you how to measure small steps at each point.

For probability distributions, the natural metric comes from the Fisher information matrix. This matrix captures how sensitive the likelihood of the data is to small changes in the parameters. If the likelihood changes rapidly when you tweak a parameter, the distribution is 'steep' there, and small moves count as large distances. If it changes slowly, the surface is flat, and moves are short.

The standard measure of separation between two distributions P and Q is the Kullback-Leibler divergence, written D_KL(P || Q). It is defined as the sum over all possible outcomes x of: the probability of x under P, multiplied by the natural logarithm of (the probability of x under P divided by the probability of x under Q).

Mathematically:
D_KL(P || Q) = the sum over all x of [ P(x) * ln( P(x) / Q(x) ) ]

When P and Q are very close to each other, this divergence behaves like a squared distance in the geometry defined by the Fisher information. Specifically, if you move from parameter value theta to theta + d_theta, the divergence is approximately:
D_KL ≈ (1/2) * d_theta_transpose * F(theta) * d_theta

Here, F(theta) is the Fisher information matrix. Its elements are the expected values of the second derivatives of the negative log-likelihood with respect to the parameters. This approximation shows that the Fisher information acts as the local ruler for measuring distances between distributions.


## A worked example

We will compare two Bernoulli distributions (coin flips). Let p be the probability of heads.

Distribution P: p = 0.5 (fair coin)
Distribution Q: q = 0.6 (biased coin)

Step 1: Calculate the Fisher information at p = 0.5.
The formula for the Fisher information of a Bernoulli distribution with parameter p is:
F(p) = 1 / (p * (1 - p))

Substitute p = 0.5:
F(0.5) = 1 / (0.5 * (1 - 0.5))
F(0.5) = 1 / (0.5 * 0.5)
F(0.5) = 1 / 0.25
F(0.5) = 4

Step 2: Calculate the Fisher information at p = 0.6.
Substitute p = 0.6:
F(0.6) = 1 / (0.6 * (1 - 0.6))
F(0.6) = 1 / (0.6 * 0.4)
F(0.6) = 1 / 0.24
F(0.6) ≈ 4.1667

Step 3: Calculate the exact Kullback-Leibler divergence D_KL(P || Q).
The formula is:
D_KL(P || Q) = sum over outcomes x in {heads, tails} of [ P(x) * ln( P(x) / Q(x) ) ]

For heads (x=1):
P(heads) = 0.5
Q(heads) = 0.6
Term for heads = 0.5 * ln(0.5 / 0.6)
ln(0.5 / 0.6) = ln(0.8333...) ≈ -0.18232
Term for heads ≈ 0.5 * (-0.18232) = -0.09116

For tails (x=0):
P(tails) = 1 - 0.5 = 0.5
Q(tails) = 1 - 0.6 = 0.4
Term for tails = 0.5 * ln(0.5 / 0.4)
ln(0.5 / 0.4) = ln(1.25) ≈ 0.22314
Term for tails ≈ 0.5 * (0.22314) = 0.11157

Sum the terms:
D_KL(P || Q) ≈ -0.09116 + 0.11157
D_KL(P || Q) ≈ 0.02041

Step 4: Calculate the quadratic approximation using Fisher information.
The approximation formula is:
Approximation = (1/2) * d_theta_transpose * F(theta) * d_theta

Here, theta = 0.5, and we move by d_theta = q - p = 0.6 - 0.5 = 0.1.
Since this is a single parameter, transpose does not change the value.
Approximation = (1/2) * F(0.5) * (0.1)^2

We calculated F(0.5) = 4 in Step 1.
Approximation = 0.5 * 4 * 0.01
Approximation = 2 * 0.01
Approximation = 0.02

Result: The exact KL divergence is approximately 0.02041. The quadratic approximation using the Fisher information at p=0.5 gives 0.02. These values are very close, confirming that for small changes in parameters, the KL divergence behaves like a squared distance weighted by the Fisher information.


## From scratch, in Python

```python
# We'll compute the Fisher information and KL divergence for a simple distribution
# First, some basic Python building blocks (if needed)
import math

# A list of possible outcomes (heads = 1, tails = 0)
outcomes = [1, 0]

# Define two distributions as lists of probabilities
# P at p=0.5, Q at p=0.6
P = [0.5, 0.5]  # probability of heads, tails
Q = [0.6, 0.4]

# Function to compute KL divergence from P to Q
def kl_divergence(P, Q):
    # P and Q are lists of probabilities, same length
    # Loop over each pair of probabilities
    total = 0.0
    for p, q in zip(P, Q):
        # skip if p is 0 (log(0) is undefined)
        if p > 0 and q > 0:
            total += p * math.log(p / q)
    return total

# Compute KL divergence
kl = kl_divergence(P, Q)
print("KL(P||Q) =", kl)

# Compute Fisher information at a given p (for a single-parameter Bernoulli)
def fisher_information(p):
    # For Bernoulli, F = 1/(p*(1-p))
    return 1.0 / (p * (1 - p))

# Compute at p=0.5
f = fisher_information(0.5)
print("Fisher at p=0.5:", f)

# Now compute the approximation: 0.5 * F * (p2-p1)^2
p1 = 0.5
p2 = 0.6
approx = 0.5 * f * (p2 - p1)**2
print("Approx KL:", approx)

```


## The same thing, with a framework

```python
import numpy as np

# Using numpy for vectorized operations
P = np.array([0.5, 0.5])
Q = np.array([0.6, 0.4])

# KL divergence: sum(P * log(P/Q))
kl = np.sum(P * np.log(P / Q))
print("KL(P||Q) =", kl)

# Fisher information for Bernoulli: use formula directly
p = 0.5
F = 1 / (p * (1 - p))
print("Fisher at p=0.5:", F)

# Approximation
approx = 0.5 * F * (0.6 - 0.5)**2
print("Approx KL:", approx)

```


## What the code does

First, we import the math module to get the natural log function. We define a list 'outcomes' but do not use it later; it is just an example of a list. Then we define two lists, P and Q, each with two elements: the probability of heads and tails. Next, we define a function 'kl_divergence' that takes two lists and returns the sum of p * log(p/q) for each pair. It uses a for loop with zip(P, Q) to pair up elements. Then we call it and print the result. Then we define a function 'fisher_information' that computes 1/(p*(1-p)), a formula we derived earlier. We call it for p=0.5. Finally, we compute the approximation using squared difference and print it.


## Common misconceptions

**Divergence is a distance** — KL divergence is a distance between distributions, so it should be symmetric and satisfy the triangle inequality.

KL divergence is asymmetric: D_KL(P||Q) is not equal to D_KL(Q||P). For example, in our coin example, D_KL(0.5||0.6) = 0.0204, but D_KL(0.6||0.5) = 0.6*ln(0.6/0.5) + 0.4*ln(0.4/0.5) = 0.6*0.1823 + 0.4*(-0.2231) = 0.1094 - 0.0892 = 0.0202. They are different. A true distance must be symmetric and satisfy the triangle inequality; KL does not. In fact: KL divergence is a divergence (a broader class of measures), not a metric. It is non-negative, zero only if the distributions are equal, but it is not symmetric and does not obey the triangle inequality. Locally, for very close distributions, it approximates a squared distance (hence the factor of 1/2).

**Fisher information is a scalar always** — Fisher information is always a single number.

When a distribution has more than one parameter (e.g., mean and variance of a Gaussian), the Fisher information is a matrix (the Fisher information matrix), where each entry measures the sensitivity of the likelihood to pairs of parameters. In fact: For a single-parameter distribution, Fisher information is a scalar: 1/(p(1-p)) for a Bernoulli. For multiple parameters, it is a matrix, typically denoted F(θ), with as many rows and columns as there are parameters.

**Natural gradient is just a fancy name for gradient descent** — Natural gradient descent is the same as ordinary gradient descent, just with a different learning rate.

Ordinary gradient descent uses the gradient of the loss with respect to parameters, which assumes a Euclidean geometry in parameter space. Natural gradient uses the gradient scaled by the inverse Fisher information matrix, which accounts for the curvature of the distribution manifold. This leads to different update directions, not just different step sizes. In fact: Natural gradient descent multiplies the gradient by the inverse Fisher information matrix, making the update step invariant to reparameterisation of the distribution. It moves in the direction that most reduces the KL divergence to the target, not the direction that most reduces the parameter difference.

**The manifold is the data** — The manifold is the set of data points I have.

The manifold is the space of all possible distributions, not the data points themselves. Data points are samples drawn from a distribution. The manifold is parameterised by the parameters of the distribution. In fact: The statistical manifold is a space where each point is a probability distribution. The data points are outcomes, not points on this manifold. You can think of the data as a set of arrows pointing to the distribution that generated them.


## In short

Information geometry treats families of probability distributions as curved surfaces (manifolds). The distance between two distributions is not the difference in their parameters, but is measured by divergences like KL divergence, which behaves locally as a squared distance scaled by the Fisher information. The Fisher information matrix is the Riemannian metric on this manifold, providing the curvature that determines how to optimise over distributions. This reframes optimisation as movement on a curved surface: the natural gradient, which uses the inverse Fisher information, is the correct steepest descent in that geometry. Remember: distance in distribution space is measured in information, not in parameter change.


## Where this is used

Where does information geometry actually show up in building AI systems? Let's go through each area.

**Large language models (LLMs).** In training an LLM, we adjust billions of parameters to minimise a loss function. The loss is a function of the model's predicted probability distribution over the next token. The space of all possible probability distributions over a vocabulary is a curved manifold. The Fisher information matrix for the full model would be enormous (billions by billions), but it can be approximated. Some recent optimisation methods, like natural gradient descent, use the Fisher information (or a diagonal approximation) to scale the gradient, so that a step in parameter space that changes the distribution a lot in information terms is made smaller. This is information geometry in action: rather than stepping in the direction that most reduces the loss in parameter space, you step in the direction that most reduces the loss in the space of distributions, respecting the curvature. Transformers also use a technique called LayerNorm that, in the 2017 paper, was applied after the residual addition (post-norm). As the book 'The Mathematics of Large Language Models' notes, this ordering mixes information across the sequence — a form of geometric mixing. In practice, developers use libraries like PyTorch, where the loss function and optimiser (e.g., Adam) are standard, but natural gradient methods are not built-in; implementing them requires computing the Fisher information, which is often approximated. A developer might write:
```python
# Pseudo-code for natural gradient update
loss = cross_entropy(predictions, targets)
grad = torch.autograd.grad(loss, model.parameters())
F = compute_fisher(model, data)  # approximated Fisher matrix
natural_grad = torch.inverse(F) @ grad
# update parameters using natural_grad
```
But in practice, full natural gradient is too expensive, so people use Kronecker-factored approximate curvature (K-FAC) or simply use Adam, which is a different kind of adaptive method.

**Small language models on devices.** When you run a model on a phone, memory and compute are limited. Quantisation, which reduces the precision of weights, changes the effective geometry of the loss landscape. Understanding the geometric structure helps in pruning and quantisation: you can retain the parameters that are most sensitive to changes in the distribution, as measured by the Fisher information. This is used in methods like Fisher-weighted pruning. On-device training (retraining on the edge) would benefit from natural gradient methods, but they are computationally heavy; approximate methods that use a diagonal Fisher are more feasible.

**Retrieval-augmented generation (RAG).** RAG retrieves documents and feeds them to an LLM. The retrieval step involves embedding queries and documents into a vector space where distances are believed to reflect semantic similarity. The choice of embedding space is a geometric choice. Information geometry comes in when you consider that the distribution of embeddings is not flat; treating it as a curved manifold could improve retrieval by using divergences (like KL divergence) between distributions of embeddings rather than simple Euclidean distance between vectors. However, in practice, most RAG systems use cosine similarity (a simple geometric measure) because it works well enough. The connection is conceptual: the space of embeddings is a manifold, and better metrics could lead to better retrieval. A developer implementing RAG would write:
```python
# Example: compute cosine similarity between query embedding and document embeddings
import numpy as np
from numpy.linalg import norm

query_emb = model.encode("What is information geometry?")
doc_emb = model.encode("A tutorial on information geometry")
cosine_sim = np.dot(query_emb, doc_emb) / (norm(query_emb) * norm(doc_emb))
```
Information geometry would suggest replacing this with a divergence that respects the curvature, but this is an active research area, not standard practice.

**Agentic systems.** Agents that plan and call tools need to make decisions under uncertainty. Bayesian inference is used to maintain a belief over states, and the update of beliefs involves distributions. Information geometry helps in approximating these distributions. For example, variational inference (which you will learn later) uses KL divergence to fit an approximate posterior; the geometry of the space of distributions affects how close you can get. In practical agent loops, a developer might use a library like `pgmpy` or `Pyro` to perform variational inference, and inside those, they use KL divergence as a loss. There is nothing specifically implemented by hand; the geometry is baked into the algorithms.

**Everyday applied ML (recommendation, vision, forecasting, fraud detection).** Many of these use probabilistic models. For instance, in fraud detection, a model might output a probability of fraud. Comparing the performance of two models is often done using log-loss (which is related to KL divergence). In recommendation systems, matrix factorisation assumes a latent space; the Euclidean geometry of that space is often used for similarity, but a more appropriate geometry might be on a manifold of distributions (e.g., multinomial distributions). In forecasting, Gaussian processes (a Bayesian method) model a distribution over functions; the covariance function defines a geometry that determines how smoothly the function varies. The Fisher information appears in experimental design, where you choose inputs to maximise the information gained about parameters. In all these, developers rarely compute information geometry explicitly; they use libraries like scikit-learn, which implement algorithms such as logistic regression (which uses a Hessian that approximates Fisher information), or PyTorch for neural networks. The key mechanism is that the geometry of the parameter space influences the optimisation dynamics, and methods that respect that geometry (like Newton's method, which uses curvature) converge faster.

In summary, information geometry is not a direct API that you call; it is a framework for understanding why certain optimisation methods work better than others, and it guides approximations in everything from LLM training to Bayesian inference. A developer who understands this can choose better optimisers or know why a certain model is unstable.


---

Read this on the web: https://basavapurushottam.com/learn-ai/information-geometry
