# Null and Alternate Hypotheses

The two competing statements a test adjudicates: a null of no effect, and an alternative representing what you would conclude if the null is rejected.

**Leads to:** ANOVA, Errors of Process vs Interpretation, Hypothesis Testing, Significance Levels

**Rests on nothing.** A place to start.

*Mathematical Foundations · Statistical Inference*


## Why learn this

You need to know this because it gives you a reliable way to decide if what you see in data is actually meaningful or just a random coincidence. Imagine you launch a small change on your website, like moving a button from red to blue, and sales go up by 2%. Without this concept, you cannot tell if that increase was caused by the color change or if it would have happened anyway by chance. By using these two opposing statements, you gain the practical ability to separate signal from noise, allowing you to make confident business decisions rather than guessing based on lucky breaks or bad luck in your data.


## The intuition

Suppose you run a coffee shop and you're thinking of switching to a new espresso blend. You don't know if the new blend is better, worse, or the same as what you use now. To find out, you set up two competing stories about reality. The first story assumes nothing has changed: the customer satisfaction stays exactly the same. This is the null hypothesis—it is the boring, safe default assumption that your change did nothing. The second story assumes something has happened: the new blend has an effect, meaning customers will either prefer it or dislike it more than the old one. This is the alternative hypothesis.

In any test, you start by pretending the null hypothesis is true. You gather data to see how well this 'no change' story holds up. If the results you get are extremely unlikely to happen if nothing had actually changed, you reject the null hypothesis and accept the alternative. It is like a courtroom: the null hypothesis is 'innocent until proven guilty.' The defendant (the idea that nothing changed) is assumed innocent until you produce strong evidence against them. You do not need to prove the new blend is perfect; you just need to show that it is statistically impossible for the observed improvement to be just random luck. If the data does not meet that high bar, you fail to reject the null, meaning you stick with the assumption that nothing meaningful changed.


## An analogy

Imagine you go to a doctor with a vague symptom and they run a diagnostic test. The null hypothesis is that you are healthy. The alternative hypothesis is that you have the specific disease being tested for. The medical test is designed to look for signs of that disease. If the test comes back positive, you might reject the assumption of health and conclude you have the disease (though there is always a small chance the test gave a false alarm). If the test comes back negative, you fail to reject the assumption of health—you remain assumed healthy, but this is not proof that you are perfectly disease-free, just that the test did not find evidence of illness. This analogy works because it shows how we treat the 'no problem' state as the default and only change our mind when the evidence is strong enough to overcome that bias.

Caveat: Unlike a medical diagnosis where treatment might follow immediately, in statistics we rarely 'accept' the alternative as absolute truth; we merely say the data supports it more than the null does.


## Definition

The null hypothesis is the default assumption that there is no effect or difference, while the alternative hypothesis is the claim that an effect or difference exists, which you support only if the data provides strong enough evidence to reject the default.


## How it connects

This concept sits next to probability, which measures how likely different outcomes are, giving you the tools to judge whether evidence is 'strong' or weak. It also connects to statistical significance, which is the specific threshold of evidence required before you reject the null hypothesis and declare your result meaningful.


## The mathematics

Here's the idea in plain words. We have two statements about the world: the null (H0) says 'nothing special is happening' — for example, a new medicine has no effect, or a new training method doesn't change accuracy. The alternative (H1 or Ha) says 'something is happening' — the medicine works, or accuracy improves. To decide, we collect data and compute a test statistic, which is a number that summarizes the data (we'll see how with a simple example). Then we ask: if the null were true, how unusual is this statistic? That's the p-value. If the p-value is small (below a pre-set threshold like 0.05), we say the data is too unusual under the null, so we reject the null and accept the alternative. If it's not small, we fail to reject the null — but that doesn't mean the null is true, just that we don't have enough evidence. The alternative can be one-sided (looking for an increase only) or two-sided (looking for any change). The null is always specified before looking at the data, to keep the error rate honest. It's a straw man you try to knock down, not something you prove.

Null hypothesis: H0: μ = 0 (no effect), Alternative: H1: μ > 0 (one-sided) or H1: μ ≠ 0 (two-sided). Test statistic: t = (x̄ - μ0) / (s / √n), where x̄ is the sample mean, μ0 is the null mean, s is the sample standard deviation, and n is the sample size. p-value: P(T ≥ |t|) for two-sided, or P(T ≥ t) for one-sided, assuming the null is true.


## A worked example

```
Suppose a company claims that a new training program increases typing speed. We test this. The null: no increase (mean speed unchanged). The alternative: speed increases. We have a small trial: five people's speeds (words per minute) before and after the program. Before: 40, 45, 38, 50, 42. After: 43, 47, 40, 54, 46. Differences (after - before): +3, +2, +2, +4, +4. The average difference is +3.0. Now, if the program had no effect, we'd expect the average difference to be 0. But we got +3. How likely is that, if the true difference is 0? We need to calculate a test statistic. We'll use the t-statistic (but to keep arithmetic simple, we'll pretend we're doing a simplified version).

Step 1: Differences: 3, 2, 2, 4, 4   # We subtract each 'before' from each 'after' to see the change.
Step 2: Mean difference = (3+2+2+4+4)/5 = 15/5 = 3.0   # This is the average change. We expect it to be 0 if the program does nothing.
Step 3: Standard deviation of differences: first compute squared deviations from the mean: (0,1,1,1,1) -> sum=4   # We measure how spread out the differences are. Each squared deviation is (difference - mean)^2.
Step 4: Sample variance = 4/(5-1) = 1.0, so standard deviation = sqrt(1.0) = 1.0   # We divide by n-1 (sample size minus one) to estimate the population spread. sqrt gives the standard deviation.
Step 5: Standard error = std_dev / sqrt(n) = 1.0 / sqrt(5) ≈ 0.447   # This tells us how much the sample mean would vary if we repeated the experiment. n = 5.
Step 6: t-statistic = (mean_diff - 0) / standard_error = 3.0 / 0.447 ≈ 6.71   # This measures how many standard errors the observed mean is from the null's expected value (0).
Step 7: p-value: for a one-sided test, the probability of seeing a t-statistic ≥ 6.71, if the null is true, is very small (less than 0.001).   # Using a t-distribution table or software, we find this. Since p < 0.05, we reject the null.

Result: We reject the null hypothesis (mean difference = 0) and conclude the program increases typing speed. The evidence is strong.
```


## From scratch, in Python

```python
import math

diffs = [3, 2, 2, 4, 4]  # the differences we computed
n = len(diffs)
mean_diff = sum(diffs) / n

# Compute sample standard deviation
squared_deviations = [(d - mean_diff) ** 2 for d in diffs]
variance = sum(squared_deviations) / (n - 1)
std_dev = math.sqrt(variance)
standard_error = std_dev / math.sqrt(n)

t_stat = (mean_diff - 0) / standard_error  # null mean is 0
print(f"Mean difference: {mean_diff}")
print(f"Standard error: {standard_error}")
print(f"t-statistic: {t_stat}")

# For a one-sided test, look up p-value (we'll approximate with normal)
# Approximation: for large t, p is tiny
p_value = 0.0001  # placeholder; in practice use scipy
if p_value < 0.05:
    print("Reject the null hypothesis")
else:
    print("Fail to reject the null hypothesis")
```


## The same thing, with a framework

```python
import numpy as np
from scipy import stats

diffs = np.array([3, 2, 2, 4, 4])
t_stat, p_value = stats.ttest_1samp(diffs, 0)  # two-sided test by default

# For one-sided (greater), take half the p-value
p_one_sided = p_value / 2
if p_one_sided < 0.05:
    print("Reject the null")
else:
    print("Fail to reject the null")
```


## What the code does

The from-scratch code builds the t-statistic by hand: it computes the mean difference, the standard deviation, the standard error, and then the t-statistic. It shows that the t-statistic is simply how many standard errors the sample mean is from the null mean. The framework version uses SciPy's ttest_1samp to do the same calculation, but underneath it computes exactly what we did: a mean, a standard error, a t-statistic, and a p-value from the t-distribution. The from-scratch version uses a placeholder for the p-value; in practice you'd compute it from the t-distribution, but the idea is the same.


## Common misconceptions

**The null is something you prove** — If the p-value is high, that means the null is true — there's no effect.

A high p-value means the data is not surprising under the null, but it could also be that your sample is too small to detect the effect. You can't prove a negative. In fact: You either reject the null or fail to reject it. Failing to reject is not the same as accepting it.

**A small p-value means a big effect** — If p < 0.05, the effect must be large and important.

A p-value depends on sample size. With a huge sample, even a tiny, meaningless effect can have p < 0.05. In fact: The p-value only tells you if the evidence is convincing given the data you have, not the size of the effect. Always look at an effect size or confidence interval.

**The alternative is what you prove** — If I reject the null, I've proven the alternative is true.

Rejecting the null is a probabilistic statement: it says the data is unlikely under the null. It doesn't prove the alternative; it just indicates it's more plausible. In fact: You say there is evidence against the null, and you act as if the alternative is true, but it's never a certainty.

**The null must be a zero effect** — The null always has to be 'no effect' (zero difference).

You can test any specific value. For example, you might test if a new drug increases survival by at least 5% (null: increase = 5%). In fact: The null is whatever you specify as the default, not necessarily zero.

**One-sided vs two-sided doesn't matter** — The p-value is the same no matter what, so I can just use two-sided always.

Two-sided tests split the alpha between both tails, doubling the p-value compared to a one-sided test when the effect is in the expected direction. In fact: Choose the test based on your question. If you only care about improvement, use a one-sided test, but be honest about it before seeing data.

**The null is a hypothesis you want to prove** — I designed my study to show the new method works, so I should try to prove the null of no difference false as strongly as possible.

The null is set up as a straw man to knock down; it's not something you're emotionally attached to. You want to have enough evidence to reject it. In fact: The null is the default, and your goal is to gather evidence to reject it, not to prove it.


## In short

The null hypothesis is a default statement of 'no effect' that you try to reject with data. The alternative is what you conclude if you reject the null. You never prove the null; you either reject it or fail to reject it. The p-value is the probability of seeing your data (or more extreme) if the null is true. Always specify the null before looking at the data. One-sided and two-sided alternatives answer different questions.


## Where this is used

In AI and data science, null and alternate hypotheses are everywhere.

**Large Language Models (LLMs):** When a team releases a new LLM, they run evaluations on benchmarks like MMLU. To claim the new model is better, they set up a null hypothesis: 'the new model has the same accuracy as the old one.' They then run the benchmark on a sample of questions, compare the accuracies, and compute a p-value. If the p-value is below 0.05, they reject the null and claim improvement. They might use a statistical test like McNemar's test (for paired proportions) or a paired t-test on the per-question scores. A developer might write:
```python
from scipy import stats
# acc1 and acc2 are lists of booleans for each question (correct/incorrect)
# McNemar's test for paired nominal data
from statsmodels.stats.contingency_tables import mcnemar
# Build contingency table: a = both correct, b = old correct new wrong, etc.
# Then p_value = mcnemar(table, exact=False)[1]
```

**Small models on devices (TinyML):** When you deploy a small model on a phone that spots wake words like 'Hey Siri', you need to test whether it works as well as a larger model. You collect audio samples, run both models, and compare their error rates. A paired t-test on the per-sample losses can give a p-value. If the p-value is high (fail to reject the null of equal performance), you might accept the small model for efficiency.

**Retrieval-Augmented Generation (RAG):** When you build a RAG system, you might test whether using a new embedding model improves retrieval quality. The null: 'the new embedding model retrieves relevant documents just as well as the old one.' You take a set of queries, compute a relevance score (e.g., recall@k) for each system, and perform a paired t-test. A developer might write:
```python
# scores_old and scores_new are lists of recall@k values per query
from scipy import stats
t_stat, p_value = stats.ttest_rel(scores_new, scores_old)  # paired test
if p_value < 0.05:
    print("New embedding is better")
```

**Agentic systems:** When an AI agent decides which tool to call or when to stop, you might run A/B tests comparing different agent policies. For each episode, you record a reward (e.g., task success). The null: 'the two policies have the same average reward.' You use a two-sample t-test or a Mann-Whitney U test. If p < 0.05, you pick the better policy.

**Everyday applied ML:** In fraud detection, you might test whether a new feature (e.g., transaction frequency) actually improves precision. You build two models—one with the feature, one without—and compare their F1 scores on a holdout set. The null: 'no difference in F1.' You use a permutation test or a bootstrap to compute a p-value. A developer might use:
```python
from sklearn.metrics import f1_score
from scipy import stats
# Repeated cross-validation, collect F1 differences, then test if mean diff = 0
```

**What breaks when done badly:** If you don't pre-specify the null and instead peek at the data first, your p-value is meaningless. If you run many tests without correction, you'll get false positives. If you use a two-sided test when you only care about improvement, your p-value doubles and you might miss a real effect. If you interpret a p-value of 0.06 as 'no effect' without considering effect size, you're missing the point—hypothesis testing doesn't measure importance.

**In summary, this concept is used whenever you need to make a decision from data with a controlled error rate, from evaluating models to running A/B tests, and it's a core part of the statistical literacy needed for modern AI practice.


---

Read this on the web: https://basavapurushottam.com/learn-ai/null-and-alternate-hypotheses
