# Repeatability Under Stochasticity

Getting stable measurements from a system that returns different outputs each run, through repetition and variance control.

**Rests on:** Sampling Distributions

**Leads to:** Four-Axis Quality, Safety, Cost and Reliability, Perturbation Robustness

*LLMs and Generative AI · LLM Evaluation*


## Why learn this

Before you can trust any comparison between two versions of an AI system, you need to know whether the difference is real or just random luck. This concept teaches you how to measure and control that randomness. It unlocks the rest of LLM Evaluation: you cannot meaningfully build benchmarks, regression gates, or evaluator drift detection until you can say how stable your measurements are.


## The intuition

Imagine you are testing two new recipes for biryani. You cook Recipe A once, taste it, and decide it's better than Recipe B. But your aunt cooked it yesterday and found it slightly salty, your cousin found it bland, and the spice blend from the market changed between batches. One tasting is not enough: the dish varies from cook to cook, and that variation can drown out the real difference between recipes. The same happens with AI models. Ask a large language model the same question twice and it may give different answers each time. That is not a bug; it is how these models work. If you compare two versions of a model by asking each one a question once, the difference you see might be the model's true quality difference — or it might be the randomness of the model's responses, or the randomness of your measurement method. Repeatability under stochasticity is the skill of designing your measurements so that the randomness does not fool you. You do this by repeating runs, measuring how much the results scatter, and reporting the scatter alongside the average. Once you know the scatter, you can judge whether the difference between two versions is bigger than the noise.


## An analogy

The darts board is a good picture. Suppose you are comparing two dart players: A and B. You let each throw one dart at a board a meter away. A hits near the bullseye, B hits the outer ring. Clear winner, right? Now suppose you let each throw twenty darts. You find that A's darts are scattered widely — one hits the bullseye, others hit the wall. B's darts, though, all land in a tight cluster just off centre. With one throw each, B lost; with twenty, you see that B is actually the more reliable player — and on average, B's darts are closer to the bullseye than A's scattered hits. The cluster width is the variance: how much a single throw tends to deviate from the average. In model evaluation, each answer is a dart. The model's 'average score' is the target. One run gives you one dart; it tells you almost nothing because it could be an outlier. Ten or twenty runs give you a cloud of darts, and from that cloud you can see where the centre is and how tight the cluster is. The analogy breaks down in one way: model outputs are not physical, independent throws — they are influenced by the prompt, the context, and the model's own parameters — so the scatter is not a fixed property like a player's skill; it changes with the task and the settings. But the central lesson holds: a single measurement is not a measurement until you know how much it wobbles.


## Definition

Repeatability under stochasticity is the practice of obtaining stable, trustworthy measurements from a system whose outputs vary randomly between runs, by repeating measurements, quantifying the spread of those results, and controlling the sources of variation you can control.


## How it connects

This concept sits at the heart of LLM Evaluation. Before you can compare two model versions, you need this: it gives you the noise floor that tells you whether a 2% difference in a standard automatic grading metric is meaningful or just random chance. It connects to several neighbouring concepts you have notes on. Deterministic validators are immune to this problem — a schema check gives the same answer every time — so you should rely on them where you can. Reference metrics like BLEU are cheaper and more reproducible than model judges, but they still face stochasticity because the model's output varies. Evaluator drift matters here too: if the judge model updates between two runs, you are not just measuring the model's randomness anymore — you are measuring a different measuring stick. The principle from your notes is that single-run comparisons are noise unless variance is quantified, and that means you cannot build honest regression gates without this concept.


## The mathematics

The branch of mathematics here is statistics, specifically descriptive statistics — the branch that summarises a bunch of numbers without trying to predict anything. You have a list of numbers (the scores from each run). Two properties of that list tell you what you need. First, the average (called the mean): the centre of the list. Second, the spread (called the standard deviation): how far, on average, each number sits from the centre. The standard deviation is the key idea. If all the scores are nearly the same, the spread is small; if they are wildly different, the spread is large.

You compute the standard deviation in a few steps:

1. Find the mean. Add all the scores together and divide by how many scores there are.
2. Find the difference between each score and the mean. Some will be positive, some negative.
3. Square each difference. Multiply each difference by itself. This turns every number into a positive value, so that a score below the mean does not cancel out a score above the mean.
4. Average those squared differences. Add them all up and divide by the count of scores. This result is called the variance.
5. Take the square root of the variance. This converts the result back into the same units as the original scores, so you can say 'the score is 85, give or take 3'. The 'give or take' is the standard deviation.

The formula for the standard deviation (read 'sigma') of a set of N scores is:

sigma = square root of ( sum of (each score minus the mean) squared, divided by N )

Here is what each part means:
- sigma (σ) is the standard deviation.
- N is the total number of scores in your list.
- x with a subscript i (x_i) is any single score in that list. The subscript i just labels which score we are looking at right now.
- mu (μ) is the mean of all the scores.
- 'sum of' means you add up the values for every score in the list.

The formula is a compressed version of the step-by-step process described above: subtract the mean, square, average, square-root.


## A worked example

```
You run an LLM evaluation five times on the same test set. The same model, the same prompts, the same questions — but the model samples its answers, so the scores vary. You get: 82, 84, 80, 86, 83.

Step 1: Mean = (82 + 84 + 80 + 86 + 83) / 5 = 415 / 5 = 83   # Add all scores together, divide by the number of runs. The centre is 83.
Step 2: Difference 1 = 82 - 83 = -1   # The first score is 1 below the mean.
Step 3: Difference 2 = 84 - 83 = 1   # The second score is 1 above the mean.
Step 4: Difference 3 = 80 - 83 = -3   # The third score is 3 below the mean.
Step 5: Difference 4 = 86 - 83 = 3   # The fourth score is 3 above the mean.
Step 6: Difference 5 = 83 - 83 = 0   # The fifth score equals the mean.
Step 7: Squares: (-1)^2 = 1, (1)^2 = 1, (-3)^2 = 9, (3)^2 = 9, (0)^2 = 0   # Square each difference so negatives and positives do not cancel.
Step 8: Average of squares = (1 + 1 + 9 + 9 + 0) / 5 = 20 / 5 = 4   # This average is called the variance — the typical squared distance from the mean.
Step 9: Standard deviation = sqrt(4) = 2   # Take the square root to get back to the original units. The 'give or take' is 2.

Result: The mean score is 83, with a standard deviation of 2. So the true score is best reported as '83 ± 2'. Any single run could easily be 2 points higher or lower than 83, just from random sampling.
```


## From scratch, in Python

```python
# We have a list of scores from repeated runs of an evaluation.
# In Python, square brackets make a list — an ordered collection that can be changed.
scores = [82, 84, 80, 86, 83]

# A function is a reusable block of code. It takes a list of numbers,
# and returns the mean (average).
def mean(numbers):
    # sum(numbers) adds all the numbers in the list.
    # len(numbers) counts how many numbers are in the list.
    return sum(numbers) / len(numbers)

# A function to compute the standard deviation.
def std_dev(numbers):
    m = mean(numbers)                # Step 1: find the mean.
    squared_diffs = []               # An empty list to hold the squared differences.
    for x in numbers:                # A for-loop repeats code for each number in the list.
        diff = x - m                 # Difference from the mean.
        squared_diff = diff * diff   # Square it (multiply by itself).
        squared_diffs.append(squared_diff)  # Add it to the list.
    # Step 2: average the squared differences.
    variance = sum(squared_diffs) / len(squared_diffs)
    # Step 3: take the square root. Import the sqrt function from the math module.
    import math
    return math.sqrt(variance)

# Run our functions on the scores.
print("Mean:", mean(scores))          # Output: Mean: 83.0
print("Std dev:", std_dev(scores))    # Output: Std dev: 2.0
```


## The same thing, with a framework

```python
In NumPy, the same calculation is a single call. NumPy computes the standard deviation for you, but make sure you know what it is doing underneath: it is doing exactly the subtract-mean, square, average, square-root steps you wrote by hand. The difference is a subtle one: NumPy defaults to dividing by N-1 (the sample standard deviation) rather than N, which is the standard deviation of the sample. For a beginner, the distinction is not critical, but it is why you might see a slightly different number from the hand calculation (for our example, numpy.std(scores, ddof=0) gives 2.0 because ddof=0 means divide by N; ddof=1 gives 2.236).
```


## What the code does

The code starts with a list of five scores. The function mean() adds them up with sum() and divides by len() to get 83. The std_dev() function loops through each score, subtracts the mean, squares the result, and collects those squared differences in a new list. It then averages them (the variance) and takes the square root. The for-loop is the core of the iteration: it looks at each number in turn and applies the same arithmetic. Importing math brings in the sqrt function. Finally, printing the results shows the mean and standard deviation. The framework version in NumPy hides the loop but does the same thing. Always question what a library call is actually doing; now you know.


## Common misconceptions

**Fixing the seed makes it fully deterministic** — If I set the random seed to 42, the model will always give the same output, so I don't need to worry about variance.

The seed controls the random number generator, but LLMs also depend on floating-point arithmetic, GPU non-determinism, and the internal state of the hosting environment. Two runs on different machines can still differ. And even if the output were identical, the evaluation measurement itself could have its own sources of randomness, such as the judge model's temperature. In fact: Fixing a seed reduces but never removes variation. Always run at least a few repetitions to measure the residual scatter.

**The mean is all you need** — I ran the evaluation three times and took the average, so now I have the true score.

Without the variance, you cannot know whether the average is stable. Three runs could all be high by luck. The average of 83 from three runs close together is different from an average of 83 from three runs spread from 70 to 96. In fact: Report the mean and the standard deviation together. The mean tells you about the centre; the standard deviation tells you how much to trust it.

**Lower temperature means no stochasticity** — If I set the temperature to 0, the model becomes deterministic, so I can compare versions with a single run.

Temperature 0 selects the most likely token, but the sampling still happens in floating-point, and there can be non-deterministic operations in the hardware and software stack. In practice, temperature 0 often yields identical outputs, but not always across different environments or versions of the framework. In fact: Treat temperature as a variance control, not a switch to deterministic. Still run multiple repetitions to confirm stability.

**The noise is the same for every task** — I measured the standard deviation on one task, so I can use that as the noise floor for all my evaluations.

The variance depends on the task, the prompt, the model, and the metric. A multiple-choice accuracy might have a tiny variance, while a free-form writing quality score might have a huge one. Using one task's standard deviation for another task's comparison is exactly the kind of assumption that produces false conclusions. In fact: Estimate the standard deviation afresh for each task and each model you compare.

**The standard deviation is the error of the model** — The standard deviation I compute from repeated runs tells me how incorrect the model is.

The standard deviation measures the spread of your measurement, not the model's accuracy. A precise measurement can still be far from the true value (biased), and a scattered measurement can still be centred correctly. Accuracy is about closeness to the truth; precision is about repeatability. In fact: Use the standard deviation as a measure of uncertainty in your measurement, and combine it with the mean to make a confidence interval, but remember a precise wrong answer is still wrong.


## In short

LLM outputs are stochastic: run the same evaluation twice and you may get different scores. To get a stable measurement, repeat the run several times, compute the mean and the standard deviation of the scores, and report the result as 'mean ± standard deviation'. Use that interval to decide whether a difference between two versions is meaningful — if the difference is smaller than the noise, you cannot trust it. Fixing seeds and setting temperatures reduce variance but never remove it. Always measure the noise; never assume it is the same across tasks.


## Where this is used

In large language models (LLMs), repeatability under stochasticity is the difference between a useful experiment and a misleading one. When an LLM is asked to produce an answer, it samples from a probability distribution over possible tokens. The randomness is controlled by the 'temperature' parameter: higher temperatures make outputs more varied, lower temperatures make them more focused but still not deterministic because of floating-point precision. A developer comparing two model versions typically writes an evaluation harness that feeds the same test set to both models, collects scores, and repeats the whole run several times. If the difference in mean scores is smaller than the standard deviation of the runs, the difference is not trustworthy. In practice, this means reporting 'Model A: 0.85 ± 0.03, Model B: 0.82 ± 0.04' rather than just the numbers. For small language models running on a phone or laptop, the randomness comes from the same sampling, but the evaluation must fit in less memory and time — so you might run the evaluation 5 times instead of 50, and you need a tighter variance to detect small differences in a smaller model. In retrieval-augmented generation (RAG), the stochasticity multiplies: the retriever may return a different set of documents on each run if it uses approximate nearest neighbour searches with random initialisation, and the generator then produces different answers from those documents. Evaluating a RAG pipeline means running the whole retrieval-plus-generation loop multiple times and measuring the variance of the final answer scores. Agentic systems that plan and call tools are the hardest case: the same prompt can trigger different tool calls, different execution orders, and different final outcomes. The variance is not just in the text, but in the actions taken, so evaluation requires repeated runs of the entire agent loop, and the variance of the task-success rate is the key number. In everyday applied machine learning — recommendation, vision, forecasting, fraud detection — the same principle holds even when the model is deterministic, because the training data is a random sample and the evaluation set may be small. A single train-test split can give a misleading accuracy; cross-validation repeats the split many times and reports the mean and standard deviation, which is exactly the pattern of repeat and measure spread. When building software with this concept, a developer uses libraries like scikit-learn's cross_val_score or writes a loop over random seeds. They set a fixed random seed to try to make the model repeatable, but they know that fixing the seed only controls the model's random number generator — it does not remove the randomness from the interaction with the evaluation set. What breaks when done badly: a team compares two prompts with a single run each, sees a 5% score difference, and changes the prompt — only to find the new prompt is actually 2% worse when measured properly. The cost is shipping a worse version to users.


---

Read this on the web: https://basavapurushottam.com/learn-ai/repeatability-under-stochasticity
