# Fisher Information

A measure of how sharply the likelihood identifies a parameter, equivalently the curvature of the log-likelihood at its maximum.

**Rests on:** Bayesian Inference and Decision Theory, Sampling Distributions

**Leads to:** Information Geometry

*Mathematical Foundations · Bayesian Methods*


## Why learn this

Fisher information explains why some questions are easy to answer from data while others are not. It tells you how much detail your observations contain about an unknown value. This matters because it helps AI systems like chatbots decide what to say next with confidence, and it guides the algorithms that train modern software. By understanding this concept, you will grasp why certain models learn quickly and others struggle, which is essential for Bayesian Methods.


## The intuition

Imagine you are trying to find the exact height of a hill by taking measurements from a distance. If the hill is a sharp, narrow peak, a single measurement tells you a lot about where the summit is because the slope changes rapidly. The 'signal' is strong and clear. If the hill is a gentle, wide mound, a measurement tells you almost nothing because the ground is flat; many different heights look nearly identical from afar. Fisher information measures exactly this sharpness, but instead of a physical hill, it applies to a parameter (an unknown value) you are trying to estimate from data.

Consider the 'likelihood function', which is simply a score that tells you how well each possible value for your parameter fits the data you have collected. If the likelihood function is sharply peaked at one specific value, that value is very well supported by the data. The Fisher information is high because small changes in the parameter cause a large drop in the fit quality, making the best value easy to identify. Conversely, if the likelihood function is flat and spread out, many different parameter values explain the data almost equally well. Here, the Fisher information is low because you cannot distinguish the true value from close alternatives. In short, Fisher information quantifies how much information your data carries about the exact value of a parameter, which directly determines how precisely you can estimate it.

To make this concrete: imagine you are estimating the bias of a coin. If you flip it 10 times and get 7 heads, the likelihood is somewhat spread out; many biases (e.g., 60% or 70%) fit reasonably well. The Fisher information is moderate. But if you flip it 10,000 times and get 7,000 heads, the likelihood function becomes a very narrow spike around 70%. The Fisher information is now extremely high because any deviation from 70% would make the observed data highly unlikely. The sharpness of that peak reflects the precision your data allows you to claim.


## An analogy

Think of trying to locate a hidden pin on a wall by observing how bright a light appears from different angles. The 'likelihood function' is like a brightness meter that tells you how well a guessed location matches the actual light intensity. Fisher information measures how quickly the brightness changes as you move your guess away from the true spot. If the light source is a tiny, intense point, moving your guess even slightly causes the brightness to drop sharply. This steep gradient means each observation gives high Fisher information, allowing you to pinpoint the location accurately. If the light source is a large, diffuse panel, the brightness changes very slowly as you move your guess. This flat response means low Fisher information; you cannot tell exactly where the center of the light is because many locations look almost identical. The analogy has a limit: Fisher information is not about the noise in your measurements, but about the shape of the 'brightness landscape' created by your model.


## Definition

Fisher information is a number that measures how sharply the likelihood function identifies an unknown parameter; it quantifies how much the score of a fit changes when you slightly adjust the parameter value away from its best estimate.


## How it connects

This concept sits beside Bayesian Methods and Information Geometry. In Bayesian Methods, which update beliefs about unknown values after seeing data, Fisher information helps determine how tight your updated confidence interval (the 'posterior distribution', which is your new belief state) will be. It also serves as the metric in Information Geometry, a field that treats probability models as shapes on a curved surface.


## The mathematics

Fisher information measures how much a set of data tells us about an unknown number we are trying to find. To understand it, we first need to look at the 'likelihood' of our data.

Imagine you have a model with one unknown parameter, which we will call theta (θ). The likelihood is a function that tells you how probable your observed data is for any given value of θ. We usually work with the logarithm of this likelihood, called the log-likelihood, because it turns products into sums, which are easier to calculate.

The log-likelihood curve has a peak. The value of θ at the very top of this peak is the 'maximum likelihood estimate'—the single best guess for the parameter based on the data. Fisher information measures how sharp that peak is.

If the peak is very narrow and steep, it means that only a tiny range of θ values produces data similar to what we observed. In this case, the data gives us strong information about θ; we know its value quite precisely. If the peak is wide and flat, many different values of θ could have produced the data. The data gives us weak information; we are uncertain about θ.

Mathematically, we measure the sharpness of the peak using the second derivative. The first derivative tells us the slope of the curve. At the very top of the peak, the slope is zero. The second derivative tells us how quickly that slope is changing as we move away from the top. A large negative second derivative means the curve bends down sharply (a narrow peak). A small negative second derivative means the curve bends gently (a wide peak).

Because the second derivative at a maximum is always negative, Fisher information is defined as the negative of this value. This ensures that Fisher information is always a positive number: larger values mean sharper peaks and more information.

For a single parameter θ, if we have N independent observations, the Fisher information I(θ) is calculated by taking the expected value of the negative second derivative of the log-likelihood with respect to θ:

I(θ) = - E [ d²ℓ(θ) / dθ² ]

Here, ℓ(θ) is the total log-likelihood for all N observations. The symbol 'E' stands for 'expected value', which means we average over all possible datasets that could have been generated by the true parameter. In practice, when we only have one dataset, we often evaluate this quantity at our best guess for θ (the maximum likelihood estimate) to get an estimate of the information.

In many standard cases, such as linear regression with Gaussian noise, the Fisher information matrix is related to the design matrix X and the noise variance σ². As shown in Passage 1, for a parameter vector β, the block of the information matrix corresponding to β is (XᵀX) / σ². This tells us that more data (larger XᵀX) increases information, while noisier data (larger σ²) decreases it.


## A worked example

We want to estimate the probability of a coin landing heads. We flip it 10 times and get 7 heads. Our parameter is p, the probability of heads.

The likelihood of seeing exactly 7 heads in 10 flips, given a probability p, is proportional to p⁷ * (1-p)³. We use the log-likelihood because it is easier to differentiate:

ℓ(p) = 7 * log(p) + 3 * log(1-p)

Step 1: Find the value of p that maximizes the log-likelihood.
The maximum occurs where the first derivative is zero. The first derivative of ℓ(p) with respect to p is:
dℓ/dp = 7/p - 3/(1-p)

Setting this to zero:
7/p = 3/(1-p)
7(1-p) = 3p
7 - 7p = 3p
7 = 10p
p = 0.7

So, the maximum likelihood estimate is p_hat = 0.7. This matches our observation (7 heads out of 10).

Step 2: Compute the second derivative of the log-likelihood.
The first derivative is 7/p - 3/(1-p). To find the second derivative, we differentiate each term with respect to p:
The derivative of 7/p (which is 7p⁻¹) is -7p⁻², or -7/p².
The derivative of -3/(1-p) (which is -3(1-p)⁻¹) is -3 * (-1)(1-p)⁻² * (-1) = -3/(1-p)². Note: The chain rule on (1-p) introduces a negative sign, and the power rule brings down a -1, so they cancel to leave a positive term in the denominator, but the original term was negative, so the result is negative.

So, the second derivative is:
d²ℓ/dp² = -7/p² - 3/(1-p)²

Step 3: Evaluate the second derivative at p = 0.7.
Substitute p = 0.7 into the formula:
d²ℓ/dp² |_{p=0.7} = -7/(0.7)² - 3/(1-0.7)²
= -7/0.49 - 3/0.09
≈ -14.2857 - 33.3333
≈ -47.619

Step 4: Calculate the Fisher information.
Fisher information is the negative of this second derivative (evaluated at the estimate):
I(p) = - (-47.619) = 47.619

Result: The estimated Fisher information at p = 0.7 is approximately 47.6. This positive number indicates the curvature of the likelihood peak at our best guess. A higher value would mean the peak is sharper, implying we have more precise information about p. With only 10 flips, this level of information is moderate; with more flips, the information would grow proportionally.


## From scratch, in Python

```python
import math

# We will compute Fisher information for a binomial example by hand.
# We have 10 flips, 7 heads. We want Fisher info at p = 0.7.

def first_derivative_log_likelihood(p, heads, total):
    # derivative of heads * log(p) + (total - heads) * log(1 - p)
    return heads / p - (total - heads) / (1 - p)

def second_derivative_log_likelihood(p, heads, total):
    # second derivative: -heads / p^2 - (total - heads) / (1 - p)^2
    return -heads / (p ** 2) - (total - heads) / ((1 - p) ** 2)

heads = 7
total = 10
p = 0.7

# Expected value of the second derivative: for a binomial, it's just the negative of the formula above.
# We compute it directly.
second_deriv = second_derivative_log_likelihood(p, heads, total)
fisher_info = -second_deriv  # negative because we want positive

print("Second derivative at p = 0.7:", second_deriv)
print("Fisher information:", fisher_info)
```


## What the code does

This code defines two functions: one for the first derivative and one for the second derivative of the log-likelihood for a binomial experiment (coin flips). We set the number of heads and total flips, and a candidate p. The first function computes 7/p - 3/(1-p), the slope. The second computes -7/p^2 - 3/(1-p)^2, the curvature. We then compute the second derivative at p=0.7 and take its negative to get the Fisher information, as in the worked example. The output is 47.62.


## Common misconceptions

**Fisher information is a property of the data** — If I have more data, Fisher information is always higher.

Fisher information depends on the parameter and the model, not on the data itself. It is defined as the expected curvature of the log-likelihood, which depends on the true parameter value, not on a specific sample. More data can help you estimate it better, but the information content per observation is fixed. In fact: Fisher information is a property of the model and the true parameter, not of a particular dataset. It measures how well the model can be expected to identify the parameter on average.

**Fisher information and variance of the estimator are the same** — A high Fisher information always means the estimator has low variance, so I can just compute Fisher information to know my estimator's variance.

The Cramér-Rao bound says the variance of any unbiased estimator is at least the inverse of Fisher information, but it does not say it equals that bound. Some estimators are inefficient and have higher variance than the bound. In fact: Fisher information gives a lower bound on the variance of unbiased estimators, but the actual variance depends on the estimator itself. The bound is achievable only if the estimator is efficient.

**Fisher information is a number you can measure from data alone** — I can collect data, compute the second derivative of the log-likelihood at the observed maximum, and that's the Fisher information.

The Fisher information is defined as an expectation over all possible data, not just the one dataset you have. The observed second derivative is an approximation and is sometimes called the observed Fisher information, but true Fisher information depends on the true parameter value. In fact: Fisher information is an expectation over the data distribution, and the observed curvature is just an estimate of it.

**Fisher information is only for Bayesian statistics** — Fisher information is all about priors and posteriors, so it's only relevant in Bayesian analysis.

Fisher information is a frequentist concept, used in maximum likelihood estimation, experimental design, and information geometry. It is also useful in Bayesian methods, but it is not exclusively Bayesian. In fact: Fisher information is a frequentist measure of information in a model, but it also appears in Bayesian contexts, such as the Jeffreys prior.


## In short

Fisher information measures how sharply the likelihood function identifies a parameter. It is the curvature of the log-likelihood at its maximum. High information means the parameter is well pinned down; low information means it is uncertain. It lower-bounds the variance of any unbiased estimator (Cramér-Rao bound) and is the Riemannian metric on the statistical manifold, making inference geometrical. It is used in optimisers like Adam and natural gradient descent, in model pruning, in Bayesian optimisation, and in giving confidence intervals.


## Where this is used

Fisher information appears everywhere in AI, even if the term is not always used. In large language models (LLMs), the training process uses gradient descent, and the step size (learning rate) is often adjusted based on the curvature of the loss landscape. This is Fisher information: the second derivative of the log-likelihood tells you how sensitive the loss is to changing a weight. Adaptive optimizers like Adam and natural gradient descent use a diagonal approximation of the Fisher information matrix to scale the learning rate per parameter. In practice, a developer using PyTorch might not write Fisher information explicitly, but the Adam optimizer they call internally uses a running average of the squared gradients, which approximates the Fisher information. In small language models that run on phones or laptops, Fisher information is used in pruning algorithms (to decide which weights can be removed without hurting performance) and in continual learning, where it tells you which parameters are crucial for previously learned tasks and should not be changed. In retrieval-augmented generation (RAG), embeddings and similarity search are used, but Fisher information informally guides how many documents to retrieve: high information means the query is specific, so fewer documents are needed; low information means you need more context. Agentic systems that plan and call tools use Fisher information in Bayesian optimisation to choose which tool to call next, balancing exploration and exploitation. In everyday applied ML, Fisher information is used in experimental design (choosing which data points to collect next), in model calibration (knowing the uncertainty of predictions), and in online learning (deciding how much to update a model from a new sample). A developer working on recommendation systems might use Fisher information to personalise user profiles, weighting new interactions based on how informative they are. In practice, a data scientist fitting a logistic regression might compute the Fisher information matrix to get confidence intervals for the coefficients: the inverse of the Fisher information is the variance of the maximum likelihood estimate. They would use a library like statsmodels or scipy, and the output gives them standard errors. This is the Cramér-Rao bound: no unbiased estimator can have a variance lower than the inverse of Fisher information. So for a developer evaluating a model, Fisher information tells them the best possible precision they can achieve. When it is done badly, ignoring Fisher information leads to overconfident models, poor uncertainty estimates, and catastrophic forgetting in neural networks. The Mathematics of Large Language Models specifically notes that the Fisher information matrix is the Riemannian metric on the statistical manifold, and that post-norm ordering in transformers mixes information across the sequence — a connection this learner’s library highlights.


---

Read this on the web: https://basavapurushottam.com/learn-ai/fisher-information
