# Reference Testing

Pinning an entire output against a stored reference so any unintended change fails loudly.

**Rests on:** Evaluation Datasets

**Leads to:** Visual Regression Testing

*Software Engineering · Software Testing*


## Why learn this

Reference testing gives you a safety net when you change how your work is done. Imagine you have a report that takes an hour to run and produces a table of numbers. You want to rewrite the code behind it to make it faster or easier to read. You need a way to be sure that, after your changes, the report still shows exactly the same numbers. Reference testing does this for you: it saves a copy of the old output and compares any new result against it. If the numbers are different, you know something went wrong. This technique is especially valuable when you are working with large datasets or complex calculations where checking every line by hand is impossible.


## The intuition

Think about keeping a sample of your favorite soup to compare against when you cook it again. You have a recipe for tomato basil soup that you love. One day, you decide to use fresh tomatoes instead of canned ones to improve the flavor. After cooking, you take a spoonful of the new soup and taste it next to the small amount of old soup you saved in a jar. You are not tasting to see if it is 'good' or 'bad'; you are tasting to see if it is *the same*. If the texture, acidity, or sweetness differs even slightly, you know your change had an unintended side effect. In software, this saved sample is called a reference. When you run your code with the same input, you compare the new output to that saved reference. If they match exactly, your changes were invisible to the data flow. If they differ, the test fails, warning you that the output has changed in a way you didn't intend.


## An analogy

Imagine a forensic accountant who keeps exact copies of all financial ledgers for a company. One day, they update the software used to generate these ledgers. To ensure the new software doesn't make a mistake, they don't just check if the numbers look right; they run the new system alongside the old one for the same month's data. They then compare every single digit in the new ledger against the old one. The old ledger is the reference. In this analogy, the accountant is you, the new software is your updated code, and the old ledger is the saved output file. The goal is not to judge the quality of the numbers, but to ensure identity between the two sets. There is a small caveat here: unlike a ledger where any difference is an error, in software, sometimes a difference is intentional—like when you decide to fix a typo in a column name. In those cases, you simply update the reference file to match the new, correct output.


## Definition

Reference testing is a method where you save the complete output of a program for a specific input as a 'golden' copy, and then automatically compare future outputs against that copy; the test passes only if every character matches exactly, and fails if anything differs.


## How it connects

Reference testing works alongside other ways of checking code. Unit tests check small, individual pieces of logic to see if they return the correct value, while integration tests check if different parts of a system work together correctly. Reference testing is similar in spirit to Visual Regression Testing, which compares images or screenshots to ensure the look of an application hasn't changed, but applies the same comparison principle to text, numbers, or data files rather than pixels.


## The mathematics

This concept draws on arithmetic, not on any advanced branch like calculus or linear algebra. The only operations are equality and inequality: is one number exactly equal to another? Reference testing is about comparison, and the mathematics of comparison is the simplest there is. When you compare two outputs, you are asking, 'Is value A the same as value B?' The answer is either yes or no. You might also ask about 'how different' — that's a measure of distance, and it uses subtraction and absolute value. For example, if your reference output says the total sales is 1000.5, and your new output says 1000.5, they are equal. If the new output says 1000.6, they are not equal — the difference is 0.1. That difference might be important or not, but reference testing cares about the simple fact of difference, not its size. This is a deliberate choice: it is strict, and it is meant to be strict. You don't want to miss a small change that could indicate a serious bug.

Let R be the reference output (a number or a collection of numbers) and N be the new output. The test passes if ∀ i, R[i] = N[i], that is, every element is exactly equal. If any i has R[i] ≠ N[i], the test fails.


## A worked example

```
You have a simple function that adds two numbers. The reference output for input (2, 3) is 5. You store that in a file. Later, you change the function to multiply instead of add. You run it with (2, 3) again.

Step 1: Reference: add(2, 3) → 5   # This is the original, known-good output.
Step 2: New code: multiply(2, 3) → 6   # The function was changed, so the output is different.
Step 3: Compare: 5 vs 6 → not equal   # The test fails, alerting you to the change.

Result: The reference test fails, telling you that the output has changed. You now know that your change had an effect, and you must decide if it is intended.
```


## From scratch, in Python

```python
# We need to compare two outputs, element by element.
# A list is an ordered collection of values, like [1, 2, 3].
reference = [1.5, 2.5, 3.5]  # This is what we stored before.
new_output = [1.5, 2.6, 3.5]  # This is what the new code produced.

# A simple function to check if two lists are identical.
def outputs_match(ref, new):
    # First, check if they have the same length.
    if len(ref) != len(new):
        return False
    # Loop over each position, comparing values.
    for i in range(len(ref)):
        if ref[i] != new[i]:
            return False  # Found a difference, so they don't match.
    return True  # All positions matched.

# Test it.
print(outputs_match(reference, new_output))  # False, because 2.5 != 2.6

# For a more realistic scenario, we store the reference in a file.
# We can read it back and compare.
# Here's how to write a reference to a file:
with open('reference_output.txt', 'w') as f:
    f.write("1.5,2.5,3.5")

# Read it back:
with open('reference_output.txt', 'r') as f:
    content = f.read()
# Convert the string of numbers to a list of floats.
stored_reference = [float(x) for x in content.split(',')]

# Now compare with new output.
print(outputs_match(stored_reference, new_output))  # False

# In practice, you'd write a test function that raises an error on mismatch.
def assert_outputs_match(ref, new):
    if not outputs_match(ref, new):
        raise AssertionError("Outputs do not match!")

# Uncomment to see it fail:
# assert_outputs_match(stored_reference, new_output)
```


## The same thing, with a framework

```python
In real projects, you'd use a testing framework like pytest. Instead of writing your own comparison function, you just use Python's built-in equality on the whole object. For example:

import pytest

def test_my_pipeline():
    reference = [1.5, 2.5, 3.5]
    new = run_my_pipeline()  # returns a list
    assert new == reference  # pytest will show the difference if it fails

For dataframes, you might use pandas.testing.assert_frame_equal, which checks exact equality and gives a detailed diff. It does the element-by-element comparison we did by hand, but in a single line.
```


## What the code does

The from-scratch code defines a function `outputs_match` that takes two lists and returns True if they are exactly equal. It first checks lengths, then loops through each index using `range` (which produces numbers 0, 1, 2...). If any pair differs, it returns False. If the loop completes without finding a difference, it returns True. Then we see how to store the reference in a text file and read it back, converting the comma-separated string into a list of floats. Finally, we wrap the comparison in an assertion that raises an error if they don't match. The framework version shows that in practice you don't need to write this from scratch — pytest and pandas provide tools that do the same comparison for you.


## Common misconceptions

**Reference testing is the same as unit testing** — Unit tests already check my functions, so I don't need reference tests.

Unit tests assert properties you choose, like 'the sum is positive'. Reference tests compare the entire output, catching unexpected changes anywhere in the output — even in parts you didn't think to check. In fact: Reference testing is a complement to unit tests. It checks the whole output against a known-good version, so it finds changes you didn't anticipate.

**Updating a reference is just part of routine development** — If the test fails, I can just update the reference file and move on.

If you update the reference without understanding why the output changed, you might be baking in a bug. The reference is your source of truth; changing it should be deliberate and reviewed. In fact: Updating a reference requires a conscious decision: you look at the diff, understand what changed, and confirm it is intended. Only then do you save the new reference.

**Reference tests are only for final outputs** — I only need to test the final result of the pipeline, not intermediate steps.

A bug in an intermediate step might cancel out later, so the final output looks fine. Pinning intermediate outputs helps you locate the cause of a change. In fact: Reference testing can be applied at any step of a pipeline. Pinning intermediate outputs gives you a finer-grained safety net.

**If my reference test passes, my output is correct** — The reference was correct once, so if the new output matches it, it's still correct.

The reference itself might have been wrong, or the input data might have changed in a way that makes the old output inappropriate. Reference testing only detects change, not correctness. In fact: A passing reference test means nothing has changed since you last verified the output. It does not guarantee the output is correct for the current input — only that it matches the stored version.

**Reference testing is only for numbers** — I can't use reference testing for text or images.

Reference testing works with any type of output as long as you can compare for equality. For text, you compare strings; for images, you compare pixel arrays. In fact: You can pin any output — text, images, JSON — and compare them for equality.


## In short

Reference testing pins an entire output against a stored reference so any unintended change fails loudly. It is a simple idea — compare and fail — but it turns refactoring and changes into safe, reviewable acts. Remember: the reference is your source of truth, updating it must be deliberate, and a passing test only means 'no change', not 'correct'.


## Where this is used

Reference testing is everywhere in data and machine learning work. Here's how it plays out in different contexts. In **large language model training pipelines**, the data preprocessing step — tokenization, filtering, deduplication — produces massive files. If someone changes the tokenizer or a cleaning rule, the entire downstream training data changes. Reference tests on a sample of the processed data catch unexpected shifts before you spend millions of rupees on GPU training. A developer writes a test that loads a small reference file of token IDs and asserts the new preprocessing produces exactly those IDs for the same input. In **small language models running on phones or laptops**, you often quantize the model to reduce size. Reference testing is used to compare the output logits (the raw scores before the softmax) of the quantized model against the full-precision model for a fixed set of prompts. The test asserts that the top-5 predictions are the same, or that the logits differ by less than a tolerance. This catches a quantization bug that might make the model respond nonsense on a specific input. In **retrieval-augmented generation (RAG)**, the retrieval step ranks documents. Reference tests pin the embeddings and the similarity scores for a fixed query-document pair. If you change the embedding model, the reference test will show that the scores changed — even if the final answer is still the same. That alerts you to review whether the new model is better or worse. In **agentic systems** that plan and call tools, reference testing is used on the tool-call sequences. For a fixed user query, the agent should call the same tools in the same order. A reference test records the sequence of function calls and arguments, and any change fails the test. This makes it safe to tweak the planning prompt — you'll know immediately if the agent starts behaving differently. In **everyday applied machine learning** — recommendation, forecasting, fraud detection — reference testing is used for feature engineering. The features you compute from raw data feed the model. If you rewrite a feature computation to be more efficient, a reference test on a small batch of raw data ensures the new features are identical to the old ones. If they aren't, you know to investigate before retraining. For all of these, the developer's workflow is the same: generate a reference output once, save it to a file, and write a test that compares every future run. The pain point is that comparing floating-point numbers directly can fail due to tiny rounding differences. That's why in practice you often use `numpy.allclose` or a tolerance. But the principle is the same: pin the output, compare, and fail loudly.


---

Read this on the web: https://basavapurushottam.com/learn-ai/reference-testing
