# Hypothesis Testing

A formal procedure for deciding whether data provide enough evidence against a null hypothesis, controlling the rate of false rejections at a chosen level.

**Rests on:** Null and Alternate Hypotheses, Sampling Distributions, Significance Levels

**Leads to:** Backtesting LLM Strategies, Cross-Validation Partitions

*Mathematical Foundations · Statistical Inference*


## Why learn this

You make decisions based on limited information every day. A colleague says their new spreadsheet formula runs faster than yours. A vendor claims their delivery service is more reliable than your current one. Without a formal method, you are just looking at two numbers and hoping for the best. Hypothesis testing gives you a controlled way to decide if a difference you see in a small set of data is a real effect or just random luck. This matters because it protects you from acting on noise. When you read in a report that "Method A scored 85 and Method B scored 82," hypothesis testing tells you whether that 3-point gap is meaningful evidence or just random variation. It is the standard tool for validating claims in AI models, medical studies, and business performance.


## The intuition

Imagine you have a bag of marbles. You cannot see inside, but you were told the bag contains exactly half red marbles and half blue marbles. You suspect this might be wrong—perhaps the bag is actually mostly red. You take out a handful to check—a sample. Suppose your handful has 8 red marbles and only 2 blue marbles. Is this enough to prove the bag is not half-and-half? Or could such a handful happen by pure chance even if the bag truly was balanced?

Hypothesis testing answers this by asking: "If the bag really were half red and half blue, how likely is it that I would draw a handful this extreme (8 red, 2 blue)?"

In our example, let us say the math shows this specific result would happen only 3% of the time by chance if the bag were truly balanced. We compare this probability to a pre-set threshold called the significance level, usually set at 5% (or 0.05). Because 3% is smaller than 5%, we decide the balanced-bag story is unlikely to be true. We reject that story.

Crucially, we never prove the bag is mostly red. We only conclude that if the bag were balanced, seeing our data would be very rare. Therefore, we act as though the bag is not balanced. The significance level controls how often we make a mistake: by choosing 5%, we accept that if the bag really were balanced, we will wrongly reject that fact 5% of the time simply due to bad luck in sampling.


## An analogy

Think of a courtroom trial. The defendant starts with a presumption of innocence. This is your null hypothesis—the default assumption that nothing unusual is happening, or there is no difference between two groups. The prosecution brings evidence (data) to try to prove guilt.

The judge does not ask, "Is the defendant actually guilty?" because absolute truth is hard to find. Instead, the judge asks: "If the defendant were truly innocent, how unlikely is it that we would see evidence this strong against them?"

This calculation is the p-value. If the evidence is so strong that it would occur by chance less than 5% of the time assuming innocence (p < 0.05), the judge rules "guilty"—in our terms, we reject the null hypothesis. However, if the evidence is weak, the judge does not rule "the defendant is innocent." The judge only says "not enough evidence to convict." This mirrors statistical practice: failing to reject the null hypothesis is not proof that the null is true; it is merely a statement that we do not have strong enough evidence to doubt it.

This analogy holds well for the logic of burdening proof, but note this limitation: in law, the jury decides the probability of guilt based on testimony. In statistics, the data is fixed and observed, and we calculate the probability of seeing that data under a specific assumption. We do not calculate the probability that the null hypothesis itself is true.


## Definition

Hypothesis testing is a method for deciding whether sample data provides strong enough evidence to reject a default assumption (the null hypothesis) in favor of an alternative, while keeping the risk of a false positive within a pre-defined limit (the significance level).


## How it connects

This concept sits next to Descriptive Statistics, which summarizes what your data looks like (e.g., averages and spreads), whereas Hypothesis Testing determines if those summaries reflect reality or sampling error. It also connects directly to Confidence Intervals, which provide a range of plausible values for a parameter rather than just a yes/no decision, offering more nuance than the binary outcome of testing.


## The mathematics

Hypothesis testing uses probability to measure how surprising your data is, assuming a specific starting assumption (called the null hypothesis) is true. The core calculation produces a number called the p-value. This number is a probability between 0 and 1. A small p-value means the observed data would be very rare if the null hypothesis were true.

The general procedure is:
1. State the null hypothesis (H0). This is the assumption you are testing, such as "the coin is fair" or "the drug has no effect."
2. Calculate a test statistic from your data. For simple cases like coin flips, this might just be the count of heads.
3. Determine the p-value. This is the probability of observing a result at least as extreme as your actual result, assuming H0 is true.
4. Compare the p-value to a significance level (alpha, α). The standard choice is 0.05 (5%).
5. If the p-value is less than alpha, you reject the null hypothesis. If it is greater than or equal to alpha, you do not reject it.

For a binomial test (like coin flips), the probability of getting exactly k heads in n flips, where the probability of heads is p, is calculated using the binomial formula:

Probability = (number of ways to choose k heads) * (p raised to the power of k) * ((1-p) raised to the power of n-k)

The "number of ways to choose k heads" is a combination, often written as "n choose k". It counts how many different orders the heads and tails could appear in.

The p-value for a one-sided test (looking for more heads than expected) is the sum of the probabilities for getting exactly k heads, plus the probability for getting k+1 heads, up to n heads.

Note: The reference passages highlight that when performing many hypothesis tests at once, standard methods can lead to false positives. The False Discovery Rate (FDR) controls the proportion of incorrect rejections among all rejections. However, for a single test, the logic above holds.


## A worked example

You have a coin. You suspect it is biased towards heads (the alternative hypothesis). The null hypothesis is that it is fair (probability of heads p = 0.5). You flip it 20 times and get 15 heads. You want to know: if the coin is truly fair, how unlikely is it to get 15 or more heads in 20 flips? That probability is the p-value. The significance level is 0.05.

Step 1: Define the parameters. Number of flips n = 20. Probability of heads under null p = 0.5. Observed heads k = 15.

Step 2: Calculate the number of ways to get exactly 15 heads in 20 flips. This is "20 choose 15". The formula for n choose k is n! / (k! * (n-k)!). 
   20! / (15! * 5!) = 15,504.
   So there are 15,504 different sequences of flips that result in exactly 15 heads.

Step 3: Calculate the probability of any one specific sequence of 20 flips. Since each flip is independent and p=0.5, the probability of any specific sequence (e.g., HHH...T) is 0.5^20.
   0.5^20 = 0.00000095367431640625.

Step 4: Calculate the probability of getting exactly 15 heads. Multiply the number of ways by the probability of one sequence:
   15,504 * 0.00000095367431640625 = 0.0147857666015625.

Step 5: The p-value is the probability of getting 15 OR MORE heads. We must sum the probabilities for 15, 16, 17, 18, 19, and 20 heads.
   We already have P(15) = 0.0147857666015625.

Step 6: Calculate P(16). Number of ways "20 choose 16" is 4,845.
   P(16) = 4,845 * 0.00000095367431640625 = 0.004620552062988281.

Step 7: Calculate P(17). Number of ways "20 choose 17" is 1,140.
   P(17) = 1,140 * 0.00000095367431640625 = 0.001087188720703125.

Step 8: Calculate P(18). Number of ways "20 choose 18" is 190.
   P(18) = 190 * 0.00000095367431640625 = 0.0001811981201171875.

Step 9: Calculate P(19). Number of ways "20 choose 19" is 20.
   P(19) = 20 * 0.00000095367431640625 = 0.000019073486328125.

Step 10: Calculate P(20). Number of ways "20 choose 20" is 1.
   P(20) = 1 * 0.00000095367431640625 = 0.00000095367431640625.

Step 11: Sum these probabilities to get the p-value:
   0.0147857666015625 + 0.004620552062988281 + 0.001087188720703125 + 0.0001811981201171875 + 0.000019073486328125 + 0.00000095367431640625 = 0.020694732666015625.

Step 12: Compare the p-value to the significance level alpha = 0.05.
   0.0207 < 0.05.

Result: The p-value is approximately 0.0207. Because this is less than 0.05, we reject the null hypothesis. There is statistical evidence that the coin is biased towards heads.


## From scratch, in Python

```python
# Hypothesis test for a coin: binomial test

# Count heads in our sample
heads = 15
n_flips = 20

# Under null, probability of heads is 0.5
p_null = 0.5

# We need to sum probabilities for 15, 16, 17, 18, 19, 20 heads
# We'll write a small function to compute (n choose k)
def choose(n, k):
    # n! / (k! * (n-k)!)
    # We'll compute it with a loop without importing math
    if k < 0 or k > n:
        return 0
    if k == 0 or k == n:
        return 1
    numerator = 1
    denominator = 1
    for i in range(1, k+1):
        numerator *= (n - k + i)
        denominator *= i
    return numerator // denominator

# Compute p-value: sum from heads to n_flips
p_value = 0.0
for k in range(heads, n_flips + 1):
    # probability of exactly k heads
    prob = choose(n_flips, k) * (p_null ** k) * ((1 - p_null) ** (n_flips - k))
    p_value += prob

# Significance level
alpha = 0.05

# Decision
if p_value < alpha:
    decision = "Reject H0"
else:
    decision = "Fail to reject H0"

print("p-value:", p_value)
print("Decision:", decision)
```


## The same thing, with a framework

```python
from scipy.stats import binom

# Number of flips and observed heads
n = 20
observed = 15

# Under null, p = 0.5
# Survival function gives P(X >= observed)
p_value = binom.sf(observed - 1, n, 0.5)  # sf(k-1) = P(X >= k)
print(p_value)
```


## What the code does

The from-scratch code starts by defining the observed data: heads = 15, n_flips = 20. We set p_null = 0.5, the probability of heads under the fair-coin null. We then define a function `choose(n, k)` that computes the number of ways to choose k items from n — this is the binomial coefficient. The function uses a loop to multiply the numerator and denominator, avoiding any external libraries. Then we loop over k from 15 to 20, and for each k we compute the probability of exactly k heads using the binomial formula: `choose(n_flips, k) * (p_null ** k) * ((1 - p_null) ** (n_flips - k))`. The `**` operator is exponentiation (raising to a power). We add these probabilities together into `p_value`. Finally, we compare `p_value` to `alpha = 0.05`. If p_value < alpha, we print 'Reject H0', else 'Fail to reject H0'. The framework version uses `scipy.stats.binom.sf`, which is the survival function (1 - CDF) — it directly gives the probability of observing 15 or more heads under the null. The `sf(observed - 1)` is a common trick because `sf(k)` gives P(X > k), so we pass `observed - 1` to get P(X >= observed).


## Common misconceptions

**p-value is the probability that the null hypothesis is true** — A p-value of 0.02 means there is a 2% chance the null hypothesis is true.

The p-value is the probability of observing data as extreme as yours, assuming the null is true. It is P(data | H0), not P(H0 | data). The probability of the null given the data requires a Bayesian prior, which is not what a frequentist p-value provides. In fact: The p-value is the probability of seeing such extreme data if the null were true. It says nothing directly about the probability that the null is true.

**Failing to reject means the null is true** — If my p-value is 0.6, that proves the coin is fair or the drug has no effect.

Failing to reject merely means the data are not strong enough to rule out the null. There could be a small effect that the study was underpowered to detect. Absence of evidence is not evidence of absence. In fact: Failing to reject means 'we cannot conclude the null is false'. It does not prove the null is true.

**The significance level is a magic threshold** — p = 0.049 is significant and p = 0.051 is not, so they are completely different results.

The 0.05 threshold is a convention, not a law of nature. The difference between 0.049 and 0.051 is virtually meaningless in terms of evidence strength. Context matters: in high-stakes decisions, you might use 0.01; in exploratory work, 0.1. In fact: The significance level is a chosen worst-case acceptable false-positive rate. It is a knob you set before the analysis, not a property of the data.

**Hypothesis testing measures effect size** — If a p-value is 0.0001, the effect must be huge.

A very small p-value can arise from a tiny effect if the sample size is huge. p-values conflate effect size and sample size. They do not tell you the magnitude or practical importance. In fact: A p-value only tells you whether the evidence against the null is strong. To know the size of an effect, look at a confidence interval or an effect size measure.

**Multiple testing is fine if each test is at 0.05** — If I run 100 tests, each with p < 0.05, each result is valid independently.

If you run 100 tests at alpha = 0.05, you expect around 5 false positives by chance even if no real effect exists. The more tests you run, the more likely you are to have at least one false positive. In fact: When running many tests, you need to correct for multiple comparisons (e.g., Bonferroni, Benjamini-Hochberg) to control the family-wise error rate or false discovery rate.


## In short

Hypothesis testing is a formal procedure for deciding whether data provide enough evidence against a null hypothesis, while controlling the chance of a false rejection. The p-value is the probability of seeing data as extreme as yours if the null were true. You compare it to a pre-chosen significance level (usually 0.05); if p-value < alpha, you reject the null. Remember: failing to reject is not the same as proving the null, a tiny p-value does not mean a big effect, and running many tests without correction inflates false positives.


## Where this is used

Hypothesis testing is everywhere in applied AI and in the design and evaluation of AI systems. Here is how it shows up:

- **Large language models (LLMs)**: When an AI lab claims that their new model outperforms the previous one on a benchmark, they report numbers — e.g., 85 vs 82. But are those numbers statistically significantly different? If they ran the benchmark once, the difference could be noise. Typically, they run the model multiple times (or on multiple seeds) and use a paired t-test or a Wilcoxon signed-rank test to ask whether the average score difference is likely to be real. The null hypothesis is 'true mean difference = 0'. The p-value tells them whether the difference is significant. In practice, many reported improvements are not tested at all, which is why you see papers with small gains that do not replicate. A developer reading a paper should ask: 'Was there a statistical test? What was the p-value?' That is hypothesis testing in action.

- **Small models on devices**: When you compress a large model to run on a phone (quantisation, pruning), you want to ensure accuracy does not degrade significantly. You cannot test on all possible inputs, so you sample inputs, run the original and compressed models, and record differences. A paired hypothesis test (e.g., McNemar's test for classification accuracy) tells you if the degradation is statistically significant or just sampling noise. If the p-value is above 0.05, you can claim the compressed model is 'statistically equivalent' (though note: failing to reject is not proof of equivalence — you need an equivalence test for that). This is critical in TinyML deployments.

- **Retrieval-Augmented Generation (RAG)**: RAG systems retrieve documents and feed them to an LLM to answer questions. Evaluating whether your RAG pipeline is better than a baseline — e.g., a different embedding model or a different chunking strategy — often involves human raters or automated metrics (like BLEU, ROUGE, or embedding similarity) on a set of queries. You compare the mean metric across queries, and you use a paired t-test to see if the difference is significant. If the p-value is not below 0.05, you cannot claim your pipeline is better. This is standard practice in IR evaluation.

- **Agentic systems (planning and tool calls)**: Agents decide which tool to call. To test whether a new prompt or planning strategy improves success rate, you run the agent on a suite of tasks with random seeds. A binomial test compares the success proportion to a baseline. For example, your baseline succeeds in 70% of tasks; your new agent succeeds in 75% of 200 tasks. Is 75% significantly better? You can use a one-proportion z-test. The p-value tells you whether the improvement is likely real or due to random seed variation. In agentic systems, because running agents is expensive, sample sizes are often small, so hypothesis testing is essential to avoid overclaiming.

- **Everyday applied machine learning** (recommendation, vision, forecasting, fraud detection):
  - **A/B testing** in recommender systems: you show half of users the new recommendation algorithm and half the old, then measure click-through rate. A two-sample t-test or chi-square test decides if the observed difference in conversion rates is significant. This is hypothesis testing in its purest form.
  - **Vision models**: when you train a new object-detection model, you test on a test set and report mAP. Comparing two models' mAP on the same test set requires a test of significance — often a paired test across images or classes.
  - **Forecasting**: if your forecast model has a lower mean absolute error than the incumbent, you need to test whether the difference is significant — often using the Diebold-Mariano test on forecast errors, which is a hypothesis test.
  - **Fraud detection**: you might test whether a new feature (e.g., transaction velocity) improves detection rate. You run a logistic regression and test the coefficient for that feature with a Wald test (which is a hypothesis test that the coefficient = 0).

**What a developer actually writes**: In Python, they will use `scipy.stats.ttest_ind` or `scipy.stats.ttest_rel` for t-tests, `scipy.stats.chisquare` or `scipy.stats.fisher_exact` for proportions, and `statsmodels` for more complex tests. The output includes a p-value, and they compare it to their pre-defined alpha (often 0.05). They also report confidence intervals. Modern practice often favours reporting effect sizes and intervals rather than a binary 'significant' — but the p-value is still the gatekeeper.

**What breaks when done badly**: If you run many tests and do not correct for multiple comparisons (e.g., testing every feature of your model), you will get false positives. If you decide your significance level after seeing the data, you are p-hacking. If you use a one-sided test when you should use two-sided, you inflate false positives. If your sample is small, your test may lack power and you may miss real effects. If your data violates assumptions (e.g., non-normal), a t-test may be invalid — you need non-parametric alternatives. In practice, many published AI results fail to replicate because researchers did not do hypothesis testing at all, or did it incorrectly.


---

Read this on the web: https://basavapurushottam.com/learn-ai/hypothesis-testing
