# Sufficient Statistics

A statistic that captures everything in the data relevant to a parameter, so that conditioning on it leaves no further information about that parameter.

**Rests on:** Descriptive Statistics, Sampling Distributions

*Mathematical Foundations · Statistical Inference*


## Why learn this

Imagine you are sorting through thousands of daily sales records to find the average price of a specific item. Writing down every single transaction is tedious and wastes space. A sufficient statistic lets you keep just the total sum of prices and the number of days sold, letting you discard the rest of the data without losing any ability to calculate that average. This compression is crucial when dealing with massive datasets or limited memory, because it allows you to focus on the specific numbers that matter for your current question rather than wading through raw details. It also helps explain why certain data aggregation methods, like taking a maximum or an average in signal processing, preserve the essential features needed to detect patterns in images or sounds.


## The intuition

Let us look at a concrete example with exam scores. Suppose you have the scores for 100 students. You want to know the class average. The 'parameter' of interest is this specific average value. To find it, you need two pieces of information: the sum of all scores and the count of students. Say the total sum is 6500 points across 100 students. The pair (6500, 100) is your sufficient statistic. From just these two numbers, anyone can calculate the average (65). If you throw away the individual list of scores and keep only this sum and count, you have lost no ability to determine that average. The individual scores do not provide any additional clue about what the average was once you already know the sum. Conversely, if you only knew the average (65) but not the sum or the count, you could not recover the total score, so the average alone is not sufficient for estimating the total. A statistic is 'sufficient' for a parameter if it captures every bit of information in the data that helps you estimate that specific parameter.


## An analogy

Think of a high-resolution photograph that you want to compress to save space. The original image has millions of pixels (the data). You want to preserve only the average brightness of a specific dark room in the photo (the parameter). You can calculate the average pixel value within that room and replace the entire room's details with just that single number. If your goal is solely to know how bright that room was, this single number is sufficient; you have discarded the rest of the image without losing any information relevant to that specific brightness value. However, if you later wanted to know the color distribution or the sharpness of objects in that room, that single average number would no longer be sufficient because it erased those other details. The analogy holds: sufficiency is always tied to a specific goal, and achieving it allows you to discard everything else.


## Definition

A sufficient statistic is a calculated summary value that contains all the information present in a dataset regarding a specific unknown quantity, meaning that once this value is known, no further data can improve your knowledge of that quantity.


## How it connects

This concept is central to statistical inference, which is the process of drawing conclusions about a whole group from a sample. It builds on descriptive statistics, such as means and medians, by explaining exactly when a summary retains all necessary information for a specific goal. It also relates directly to hypothesis testing, where 'power' refers to the ability of a test to correctly detect an effect if one truly exists; using sufficient statistics ensures you do not waste this power on irrelevant data details. In deep learning, it connects to pooling layers in convolutional neural networks, which reduce image data size by summarizing regions into single values that are sufficient for detecting the presence of features like edges or textures.


## The mathematics

A sufficient statistic is a summary of data that holds all the information needed to estimate an unknown parameter. To understand this, we must first define the terms.

The **parameter** is the true value we want to know about a population. For example, if we are studying a fair six-sided die, the parameter is the average value of a single roll. Since the faces are 1, 2, 3, 4, 5, and 6, the average is (1 + 2 + 3 + 4 + 5 + 6) divided by 6, which equals 3.5.

The **data** consists of the observations we collect. Let us say we roll the die N times. We record the results as a list of numbers: x_1, x_2, ..., x_N. Here, x_1 is the result of the first roll, x_2 is the result of the second roll, and so on.

A **statistic** is any number calculated from the data. Common statistics include the sum of the rolls or the average of the rolls. A statistic is called **sufficient** for a parameter if knowing the value of that statistic gives us exactly as much information about the parameter as knowing the entire list of raw data does.

There are two ways to see why this matters.

First, consider the definition based on probability. The Fisher-Neyman factorization theorem states that a statistic T(X) is sufficient for a parameter theta if the probability of observing the specific data X can be split into two parts:
1. A part that depends only on the data itself, not on the parameter.
2. A part that depends on the parameter only through the statistic T(X).

Mathematically, this looks like:
P(data | theta) = h(data) * g(theta, T(data))

Here, P(data | theta) is the probability of seeing our specific set of rolls given the true average theta. The function h(data) captures features of the data that do not help us find theta (for example, the order in which the dice landed). The function g(theta, T(data)) captures all the information about theta. If we can write the probability this way, then no other feature of the data contains extra information about theta.

Second, consider the intuitive meaning. If a statistic is sufficient, then once you know its value, the specific arrangement of the raw data becomes irrelevant for estimating the parameter. The raw data and the parameter are effectively independent once the sufficient statistic is known.

Let us look at the example of estimating the mean of a fair die. We roll the die N times. The sum of the rolls, S = x_1 + x_2 + ... + x_N, is a sufficient statistic for the mean. Why? Because the sample mean (the estimate of the population mean) is calculated as S divided by N. If we know S and we know N, we know the sample mean exactly. We do not need to look at x_1, x_2, etc., individually to calculate the mean. The sum contains all the information about the location of the distribution that is available in the data.

Note: This concept applies strictly when the underlying model (the probability distribution) is known. For distributions in the exponential family, such as the Gaussian or Bernoulli distributions, there is a direct link between the form of the distribution and its sufficient statistics.


## A worked example

We will verify that the sum of rolls is a sufficient statistic for the mean of a fair six-sided die.

**Setup:**
- The die has faces: 1, 2, 3, 4, 5, 6.
- The true population mean (theta) is 3.5.
- We roll the die N = 5 times.
- We compare two different datasets that produce different sums.

**Dataset A:**
The rolls are: 2, 6, 4, 3, 5.

Step 1: Calculate the sum of Dataset A.
Sum_A = 2 + 6 + 4 + 3 + 5
Sum_A = 8 + 4 + 3 + 5
Sum_A = 12 + 3 + 5
Sum_A = 15 + 5
Sum_A = 20.

Step 2: Calculate the sample mean from Sum_A.
Sample Mean_A = Sum_A / N
Sample Mean_A = 20 / 5
Sample Mean_A = 4.

**Dataset B:**
The rolls are: 5, 1, 3, 6, 2.

Step 3: Calculate the sum of Dataset B.
Sum_B = 5 + 1 + 3 + 6 + 2
Sum_B = 6 + 3 + 6 + 2
Sum_B = 9 + 6 + 2
Sum_B = 15 + 2
Sum_B = 17.

Step 4: Calculate the sample mean from Sum_B.
Sample Mean_B = Sum_B / N
Sample Mean_B = 17 / 5
Sample Mean_B = 3.4.

**Verification of Sufficiency:**
We ask: Does knowing the individual rolls provide more information about the population mean than knowing just the sum?

For Dataset A:
- We know the sum is 20.
- The estimate for the mean is 20 / 5 = 4.
- If we look at the individual rolls (2, 6, 4, 3, 5), do they change this estimate? No. The formula for the mean uses the sum. The specific values 2, 6, 4, 3, 5 only matter because they add up to 20. Any other set of five rolls that also adds up to 20 (for example, 1, 1, 1, 1, 16 is impossible for a die, but hypothetically) would yield the same mean estimate.

For Dataset B:
- We know the sum is 17.
- The estimate for the mean is 17 / 5 = 3.4.
- The individual rolls (5, 1, 3, 6, 2) do not provide any additional way to calculate the mean. They are only useful insofar as they contribute to the sum of 17.

**Conclusion:**
The sum is a sufficient statistic for the mean because the calculation of the mean depends entirely on the sum and the count N. Once the sum is known, the specific order or identity of the individual rolls does not change our estimate of the parameter theta. The data provides no extra information about theta beyond what is contained in the sum.


## From scratch, in Python

```python
# We don't need any special libraries for this simple example.
# We'll just compute the sum of a list of numbers.

# A list is an ordered collection of items, like a list of test scores.
scores = [85, 92, 78, 90, 88]

# To find the average, we sum all the scores and divide by how many there are.
# The sum() function adds up all numbers in the list.
total = sum(scores)
count = len(scores) # len() gives the number of items in the list.
average = total / count

print("Total:", total)   # Output: 433
print("Count:", count)   # Output: 5
print("Average:", average) # Output: 86.6

# The total is a sufficient statistic for the average.
# Given the total and count, we can compute the average, and we don't need the original scores.
```


## What the code does

This code uses two basic Python building blocks: a list and two built-in functions. A list is a collection of values, like scores. The sum() function adds up all the numbers in the list, giving us the total. The len() function tells us how many items are in the list. To compute the average, we divide the total by the count. This is exactly the sufficiency idea: we could have just passed the total and count to someone else, and they would know the average without seeing the original scores.


## Common misconceptions

**Sufficient statistic is the only way to summarise data** — If I know the average, I know everything about the data for any purpose.

The average is sufficient only for the mean. It tells you nothing about the spread, skewness, or outliers, which may be the parameters you care about in other contexts. In fact: A sufficient statistic is only sufficient for a specific parameter or set of parameters. For the variance, you would need the sum of squares as well.

**Sufficient statistic must be a simple sum** — Statistics can only be sums or counts, not something more complex like a median or a learned embedding.

Sufficient statistics can be complex functions, like a vector of a neural network's last hidden layer, which captures enough information for classification. In fact: Any function of the data that preserves all information about the parameter can be sufficient, no matter how complex.

**If a statistic is sufficient, it is also minimal** — All sufficient statistics compress the data equally well.

Some sufficient statistics retain extra useless information. Minimal sufficiency is about the strongest compression that still preserves all relevant information. In fact: A minimal sufficient statistic is a sufficient statistic that is a function of every other sufficient statistic, so it contains no superfluous information.

**Sufficiency means the data are independent** — If we have a sufficient statistic, the original data are irrelevant and can be discarded entirely.

The data are still useful for other analyses or for checking model assumptions; sufficiency only says that for a specific parameter, the statistic alone is enough. In fact: You can discard the data if you only care about that parameter, but if you later need to estimate a different parameter, you may need the original data.


## In short

A sufficient statistic is a single number (or a set) that captures all the information in the data about a parameter. It allows you to compress data without losing inferential power. Use it to summarise data efficiently for estimation and hypothesis testing, and as a principle in deep learning for operations like pooling and context summarisation in models.


## Where this is used

In large language models (LLMs), sufficient statistics appear in the attention mechanism, where instead of using all of a sequence's tokens to predict the next one, a model often uses a summary vector (like the mean of token embeddings) as a sufficient statistic for the context. This reduces memory and computation. In small models running on devices, sufficiency is used in quantization: instead of storing all weight values, you store their mean and range as sufficient statistics for the weights, allowing compression. In retrieval-augmented generation (RAG), when you embed documents, you often compute a single vector for a document (e.g., by averaging word embeddings) as a sufficient statistic for the document's meaning, so you don't have to store the entire text for similarity search. In agentic systems, when an agent plans, it doesn't collect all possible observations; it computes a sufficient statistic of the environment's state to decide the next action. In everyday applied ML, pooling in CNNs (as in your notes) is a sufficiency argument: a max-pool over a small region captures the most salient feature in that region, and other values are irrelevant for detection. For developers, this means knowing when to summarise data without losing predictive performance: for example, in time-series forecasting, using the cumulative sum of a signal can be sufficient for the trend parameter, saving memory. A developer writes code to compute these statistics (like the sum() we did) and designs models that take such statistics as inputs, rather than raw data. Done badly, they discard data that were not sufficient and lose crucial information, leading to poor model performance.


---

Read this on the web: https://basavapurushottam.com/learn-ai/sufficient-statistics
