# Fitting Distributions

Choosing a parametric family and estimating its parameters from data, then checking whether the fit is adequate rather than assuming it.

**Rests on:** Empirical CDF, Sampling Distributions

**Leads to:** Bayesian Inference and Decision Theory, Drift Detection, Membership Inference, Naive Bayes

*Mathematical Foundations · Probability Theory*


## Why learn this

Fitting distributions is the skill that lets you summarize a pile of data with just two or three numbers, turning a chaotic list of measurements into a clear picture of what usually happens and what is rare. Imagine you collect the wait times for 100 buses: listing all 100 numbers tells you little, but saying "the waits are roughly normally distributed with an average of 12 minutes and a spread of 4 minutes" gives you immediate power. You can predict that most buses will arrive within a predictable window, flagging any delay as an anomaly rather than noise. This skill is the foundation for understanding uncertainty in your work. Without it, every new set of data feels like starting from scratch. With it, you recognize patterns—whether they are symmetric bell shapes or lopsided curves—and can make informed decisions based on what the data actually shows, not just what it looks like at a glance.


## The intuition

Think of a distribution as a smooth, idealized curve that represents the pattern underlying your messy real-world data. Real data is jagged; every measurement is unique. But if you plot how often different values occur, a shape emerges. Fitting a distribution is like trying to trace that emerging shape with a pre-made stencil from a book of standard curves.

You do not draw the curve from scratch. Instead, you pick a family of curves—for example, the "normal" curve, which is symmetrical and bell-shaped—and then you adjust its "knobs." These knobs are called parameters. For a normal curve, the two knobs are the center (the mean) and the width (the standard deviation). You twist these knobs until the stencil hugs your data as closely as possible.

How do you know which knob settings are the best? The most common method is called "maximum likelihood." This sounds complex, but it simply means: choose the knob settings that make the actual data you have the most probable outcome. If your data has many values near 10 and few near 20, you adjust the curve so its peak sits near 10. You are looking for the scenario where your observed data was most likely to be generated.

Once you have set the knobs, you must verify the fit. A curve might look good in the middle but fail at the edges. This is where you check for "goodness-of-fit." You ask: does this curve honestly represent the entire dataset, including the rare extremes? To check this, you often use diagnostic plots—visual comparisons between your actual data counts and the counts predicted by your curve—or statistical tests that quantify the mismatch.

Crucially, not all data fits a bell shape. Sometimes data is "skewed," meaning it is lopsided, with a long tail stretching out to one side (like income data, where most people earn modest sums but a few earn vastly more). If your data is skewed, a normal curve will never fit well, no matter how you twist the knobs. You must then try a different family of curves, one designed for lopsided shapes.


## An analogy

Imagine you are trying to capture the shadow of a complex object on a wall using only a few light bulbs. The data is the actual shape of the object casting the shadow. Fitting a distribution is like choosing a handful of standard light sources—perhaps a round spotlight, a rectangular panel, or a focused beam—and adjusting their brightness and position until the combined light creates a silhouette that matches the object as closely as possible.

You don't paint the shadow pixel by pixel; you adjust the positions (parameters) of your fixed light shapes (distribution families) to approximate the result. If the object is a perfect sphere, a single round spotlight might suffice. If the object is an L-shape, you might need two spotlights at different angles. The process involves adjusting the knobs until the approximation is as close as possible, and then checking from different angles (goodness-of-fit) to ensure no parts of the object are hidden or misrepresented by the lights.

The analogy breaks down slightly because, unlike light bulbs, you cannot change the shape of the distribution family itself; you can only move it and stretch it. If your object is a starburst shape, no amount of moving standard circles or rectangles will make them fit perfectly.


## Definition

Fitting a distribution involves selecting a standard mathematical curve from a known family, adjusting its parameters to maximize the probability that this curve would produce your observed data, and then verifying that the fit adequately captures the data's structure across all ranges, not just the center.


## How it connects

This concept sits at the heart of understanding how samples relate to populations. It connects directly to **Sampling Distributions**, which describes how the parameter estimates you find (like the mean) would vary if you collected many different samples from the same population. It also connects to **Concentration Inequalities**, which provide mathematical guarantees about how closely your sample's average will likely stay to the true population average, helping you trust your fitted parameters.


## The mathematics

Fitting a distribution is the process of choosing a mathematical model that describes how likely different outcomes are in your data. The most common method is called **maximum likelihood estimation**. This method asks: "If I assume my data comes from a specific type of curve (like a bell curve), which version of that curve makes the actual data I collected look the most probable?"

To answer this, we use a tool called the **likelihood function**. The likelihood is not a probability itself; it is a score. It tells you how well a specific set of parameters (the numbers that define the shape and position of your curve) explains the data you have.

Here is how to build that score step by step:

1. **Define the model.** You choose a family of distributions, such as the normal distribution (the bell curve). This distribution is defined by two parameters: the mean (μ, read 'mu'), which sets the center, and the standard deviation (σ, read 'sigma'), which sets the width.

2. **Evaluate each data point.** For every single observation in your dataset, you calculate its probability density using the formula for the normal distribution with your chosen μ and σ. Let us call this value f(x_i; θ). Here, x_i is the i-th data point, and θ (read 'theta') represents the parameters μ and σ together.

3. **Combine the observations.** Because the data points are independent, the likelihood of seeing the entire dataset is the product of the individual densities. If you have n data points, the likelihood L(θ) is:

L(θ) = f(x_1; θ) × f(x_2; θ) × ... × f(x_n; θ)

This can be written more compactly using product notation:

L(θ) = ∏_{i=1}^{n} f(x_i; θ)

The symbol ∏ (capital pi) means "multiply the terms that follow, for i starting at 1 and ending at n."

4. **Switch to log-likelihood.** Multiplying many small numbers (probabilities are always between 0 and 1) creates extremely tiny results that computers struggle with. To fix this, we take the natural logarithm of the likelihood. This gives us the **log-likelihood**, denoted ℓ(θ) (read 'ell of theta').

Because the logarithm is a monotonically increasing function, the value of θ that maximizes the log-likelihood is exactly the same as the value that maximizes the likelihood. The log-likelihood turns the product into a sum:

ℓ(θ) = ln(L(θ)) = ∑_{i=1}^{n} ln(f(x_i; θ))

The symbol ∑ (capital sigma) means "add the terms that follow, for i starting at 1 and ending at n."

5. **Find the maximum.** You search for the specific values of μ and σ that make ℓ(θ) as large as possible. These values are called the **maximum likelihood estimates** (MLE).

For the normal distribution, there is a direct formula for these estimates:

The estimate for the mean, μ̂ (read 'mu hat'), is the sample mean:
μ̂ = (1/n) ∑_{i=1}^{n} x_i

This means you add up all your data points and divide by the count n.

The estimate for the variance, σ̂² (read 'sigma squared hat'), is the average of the squared differences from the mean:
σ̂² = (1/n) ∑_{i=1}^{n} (x_i - μ̂)²

Note that we divide by n here, not n-1. This is a key distinction between the maximum likelihood estimate and the unbiased sample variance often taught in introductory statistics.

**Alternative: Method of Moments**
Another way to fit a distribution is the **method of moments**. Instead of maximizing probability, you match the theoretical properties of the distribution to the observed properties of the data.

For example, for the normal distribution:
- Set the theoretical mean E[X] equal to the sample mean (1/n) ∑ x_i.
- Set the theoretical second moment E[X²] equal to the sample second moment (1/n) ∑ x_i².

You then solve these equations for μ and σ. This method is simpler to calculate but often less efficient than maximum likelihood, meaning it requires more data to achieve the same accuracy.

**Checking the Fit**
After finding the best parameters, you must verify that the model actually fits the data. You do this by:
1. Plotting a histogram of your data and overlaying the curve defined by your estimated μ and σ.
2. Using statistical tests like the Kolmogorov-Smirnov test or Chi-square test to quantify the difference between the observed data and the predicted distribution.


## A worked example

We have collected the waiting time (in minutes) at a bus stop for 10 buses: [2, 3, 1, 4, 2, 3, 5, 2, 4, 3]. We hypothesize that the waiting times follow a normal distribution. We want to estimate the mean (μ) and standard deviation (σ) using maximum likelihood.

**Step 1: Calculate the sample mean (μ̂)**
Add all the waiting times together:
2 + 3 + 1 + 4 + 2 + 3 + 5 + 2 + 4 + 3 = 29

Divide by the number of observations (n = 10):
μ̂ = 29 / 10 = 2.9 minutes.

**Step 2: Calculate the squared deviations from the mean**
For each data point x_i, subtract the mean (2.9) and square the result:
- For 2: (2 - 2.9)² = (-0.9)² = 0.81
- For 3: (3 - 2.9)² = (0.1)² = 0.01
- For 1: (1 - 2.9)² = (-1.9)² = 3.61
- For 4: (4 - 2.9)² = (1.1)² = 1.21
- For 2: (2 - 2.9)² = (-0.9)² = 0.81
- For 3: (3 - 2.9)² = (0.1)² = 0.01
- For 5: (5 - 2.9)² = (2.1)² = 4.41
- For 2: (2 - 2.9)² = (-0.9)² = 0.81
- For 4: (4 - 2.9)² = (1.1)² = 1.21
- For 3: (3 - 2.9)² = (0.1)² = 0.01

**Step 3: Sum the squared deviations**
0.81 + 0.01 + 3.61 + 1.21 + 0.81 + 0.01 + 4.41 + 0.81 + 1.21 + 0.01 = 12.9

**Step 4: Calculate the maximum likelihood estimate for variance (σ̂²)**
Divide the sum of squared deviations by n (not n-1, because this is the MLE):
σ̂² = 12.9 / 10 = 1.29

**Step 5: Calculate the maximum likelihood estimate for standard deviation (σ̂)**
Take the square root of the variance:
σ̂ = √1.29 ≈ 1.136 minutes.

*(Note: If you were calculating the unbiased sample variance for a t-test, you would divide by n-1=9, giving 1.433, and σ≈1.197. However, for maximum likelihood fitting of the distribution itself, we use n=10.)*

**Step 6: Check the fit**
We now have a fitted normal distribution with mean 2.9 and standard deviation 1.136.
To check if this fits well:
1. Draw a histogram of the data [2, 3, 1, 4, 2, 3, 5, 2, 4, 3].
2. Overlay the bell curve defined by μ=2.9 and σ=1.136.
3. Visually inspect if the curve follows the bars of the histogram.
4. Optionally, perform a Kolmogorov-Smirnov test. This test compares the cumulative distribution of your data to the cumulative distribution of your fitted model. A small difference (D statistic) and a high p-value (e.g., > 0.05) indicate that the normal distribution is a plausible model for this data.

**Result:**
The waiting times are modeled as normally distributed with mean 2.9 minutes and standard deviation 1.136 minutes.


## From scratch, in Python

Let's implement maximum likelihood estimation for a normal distribution from scratch in Python. We'll need to know about lists, functions, and the math module.

```python
import math

# Our data: waiting times at a bus stop
data = [2, 3, 1, 4, 2, 3, 5, 2, 4, 3]
n = len(data)  # number of data points

# Step 1: compute the sample mean
total = 0
for x in data:
    total = total + x
mean = total / n  # = 29/10 = 2.9

# Step 2: compute the sample variance (MLE uses n, not n-1)
sum_sq_dev = 0
for x in data:
    sum_sq_dev = sum_sq_dev + (x - mean) ** 2
variance = sum_sq_dev / n  # = 12.9/10 = 1.29
std_dev = math.sqrt(variance)  # ≈ 1.136

# Step 3: compute the log-likelihood for the normal distribution
# The log-likelihood of the whole dataset is the sum of the logs of the individual densities.
# For a normal density, log(f(x)) = -0.5*log(2*pi) - log(std_dev) - (x-mean)^2/(2*std_dev^2)

def normal_logpdf(x, mu, sigma):
    return -0.5 * math.log(2 * math.pi) - math.log(sigma) - (x - mu) ** 2 / (2 * sigma ** 2)

log_likelihood = 0
for x in data:
    log_likelihood = log_likelihood + normal_logpdf(x, mean, std_dev)

print("Mean:", mean)
print("StdDev:", std_dev)
print("Log-likelihood:", log_likelihood)

# Step 4: check the fit by computing the empirical CDF and comparing to the theoretical CDF
# We'll just print a simple diagnostic: the maximum difference between sorted data and the normal CDF.
# This is a crude version of the Kolmogorov-Smirnov statistic.

def normal_cdf(x, mu, sigma):
    # Using the error function to compute the CDF
    return 0.5 * (1 + math.erf((x - mu) / (sigma * math.sqrt(2))))

# Sort data
sorted_data = sorted(data)

max_diff = 0
for i, x in enumerate(sorted_data):
    # empirical CDF at x is (i+1)/n
    emp_cdf = (i + 1) / n
    theo_cdf = normal_cdf(x, mean, std_dev)
    diff = abs(emp_cdf - theo_cdf)
    if diff > max_diff:
        max_diff = diff

print("Max KS-type difference:", max_diff)
```


## The same thing, with a framework

In practice, you would use a statistics library like SciPy or statsmodels. For example, using `scipy.stats`:

```python
import numpy as np
from scipy import stats

data = np.array([2, 3, 1, 4, 2, 3, 5, 2, 4, 3])

# Fit a normal distribution to the data using maximum likelihood
mu, sigma = stats.norm.fit(data)  # returns (mean, std_dev) with MLE (sigma is sqrt of variance with n divisor)

print("Fitted mean:", mu)
print("Fitted sigma:", sigma)

# Compute log-likelihood
log_lik = stats.norm.logpdf(data, mu, sigma).sum()
print("Log-likelihood:", log_lik)

# Perform a Kolmogorov-Smirnov test against a normal distribution with fitted parameters
ks_stat, p_value = stats.kstest(data, 'norm', args=(mu, sigma))
print("KS statistic:", ks_stat)
print("P-value:", p_value)
```

Underneath, `stats.norm.fit` computes the sample mean and the sample standard deviation using the maximum likelihood formula (dividing by n, not n-1). The `kstest` function computes the maximum difference between the empirical CDF and the fitted normal CDF, just like the manual calculation above. This library does all the number-crunching for you, but now you know exactly what it is doing.


## What the code does

First, we define our data as a list of numbers. We use `len(data)` to get its length. Then we compute the mean by summing all elements and dividing by n. The for loop `for x in data:` iterates over each element, and we accumulate the total. After that, we compute the variance by summing squared deviations. We define a function `normal_logpdf` that takes an x value, mean, and sigma, and returns the log-density. We sum the log-densities for all data points to get the log-likelihood. Then we compute a crude goodness-of-fit measure by comparing the empirical CDF (based on the sorted data) to the theoretical normal CDF. The `math.erf` function is used for the normal CDF. The maximum difference (a KS-like statistic) tells us how far the data deviates from the fitted curve.


## Common misconceptions

**The fitted distribution is the true distribution.** — I estimated the parameters, so now I know the exact distribution of the data.

Estimates are uncertain. There is sampling variability; if you collected another sample, you would get a different estimate. Also, the family itself might be wrong. In fact: The fitted distribution is an approximation. You must quantify uncertainty (e.g., using confidence intervals) and verify the fit with diagnostics.

**Maximum likelihood is always the best method.** — Maximum likelihood is the only correct way to fit distributions.

MLE has nice properties, but it can be biased for small samples, and it is not always robust to outliers. The method of moments is simpler and sometimes better in practice. In fact: MLE is the default, but you should understand its assumptions and consider alternatives when data is small or has anomalies.

**A well-fitting distribution is good for extrapolation.** — If the fit matches my data, I can safely predict beyond the range of my data.

A distribution that fits the observed middle may have a tail shape that is completely wrong. Extrapolation depends critically on the family chosen, not just the fit to the current data. In fact: Always be careful with extrapolation. Use goodness-of-fit tests, but also understand the process that generates the data to ensure the family is physically plausible.


## In short

Fitting a distribution is choosing a curve with a few parameters to summarize your data. The default method is maximum likelihood, which picks parameters that make the data most probable. Always check the fit with diagnostic plots and hypothesis tests. A wrong family will mislead extrapolation.


## Where this is used

Fitting distributions is everywhere in AI and data science. Let's go through each area you asked about.

**Large Language Models:** LLMs output probabilities over tokens. When sampling text, the model uses a probability distribution (often a categorical distribution over the vocabulary). The softmax function turns raw scores into a distribution. But fitting distributions comes in during training: the loss function (cross-entropy) is derived from the likelihood of the training data under the model's predicted distribution. In other words, training is maximum likelihood estimation applied to the model's output distribution. The parameters are the neural network weights, and the fitting procedure (gradient descent) adjusts them to maximize the likelihood of the training sequences.

**Small models on devices:** When you run a model on a phone or a smartwatch, you need it to be fast and small. Fitting distributions is used in model compression: you might fit a simple distribution (like a Gaussian) to the activations of a neural network and then quantize the values to reduce memory. Also, in on-device personalization, you might fit a distribution to the user's usage patterns to predict their next action (e.g., which app they will open).

**Retrieval-Augmented Generation (RAG):** In RAG, you retrieve relevant documents and then feed them to an LLM. The retrieval step often involves ranking based on similarity. That similarity score can be modeled as coming from a distribution. For example, you might fit a Gaussian distribution to the similarity scores of truly relevant documents, and then use that to decide a threshold for retrieval. Also, RAG systems need to estimate the uncertainty of the retrieval: if the top document's score is not significantly higher than the rest (judged by the fitted distribution), the system might fall back to a different strategy.

**Agentic systems:** Agents that plan and call tools need to model uncertainty about the outcomes of their actions. For example, if an agent is deciding whether to search the web or query a database, it might estimate the distribution of response times or the probability of success. Fitting distributions to historical response times helps set timeouts and decide when to abandon a tool call. Also, in reinforcement learning (a type of agentic system), value functions are often represented as distributions (e.g., Gaussian) rather than single numbers.

**Everyday applied ML:** This is where fitting distributions is most directly used. In **fraud detection**, transaction amounts or frequencies are modeled as distributions; anomalies are data points that fall in the low-probability tail. In **forecasting** (e.g., sales or demand), you fit a distribution to historical data and then predict future quantiles to handle uncertainty. In **recommendation systems**, user ratings are modeled as a distribution (e.g., a Gaussian) and you rank items by the expected rating. In **vision** (object detection), you might fit a distribution to the size of bounding boxes to filter out unrealistic detections. In **speech** recognition, audio features are often modeled with Gaussian mixtures, which is exactly fitting a mixture of normal distributions.

**What a developer writes:** In Python, they usually use `scipy.stats` to fit distributions (as shown above) or `statsmodels` for more complex models. In MATLAB, they use the `fitdist` function, which fits a distribution using maximum likelihood. The library does the heavy lifting, but the developer must choose the family, fit it, and then check the fit with `qqplot` or a goodness-of-fit test. Done badly—for example, fitting a normal distribution to heavily skewed data without checking—leads to poor predictions and unreliable uncertainty estimates. Done well, it turns data into actionable insight.

**In practice, fitting distributions is also used for synthetic data generation.** You might fit a distribution to a real dataset, then sample from that fitted distribution to create synthetic data for testing or data augmentation. This is common in medical imaging or financial modeling where data is scarce.


---

Read this on the web: https://basavapurushottam.com/learn-ai/fitting-distributions
