# Evaluation Datasets

Curated sets of inputs with expected properties or outputs, forming the fixed reference against which changes are compared.

**Leads to:** Benchmarking, Black-Box vs White-Box Assessment, BLEU and Reference Metrics, Deterministic Validators, Evaluator Drift, Gold and Adversarial Sets, LLM Safety Benchmarks, Minimum Evaluable Product, Offline vs Online Evaluation, Production Feedback Loops, Promptfoo and LightEval, Prompt Optimization, Reference Testing, Regression Gates, ROUGE and Human Eval, Skill Testing and Evals, Synthetic Eval Data, Trust Rating Scales

**Rests on nothing.** A place to start.

*LLMs and Generative AI · LLM Evaluation*


## Why learn this

You cannot manage what you cannot measure. If you update a customer service bot, a fraud detection tool, or a coding assistant, you need proof that the new version works as well as the old one did—not just on random samples, but against the exact same tests every time. Without a fixed reference point, you are flying blind. You might think a model improved because it handled one easy question better, while ignoring that it failed three crucial ones it used to answer correctly. A stable dataset gives you that proof. It allows you to catch mistakes before they reach users, compare different versions fairly, and verify that improvements in one area do not accidentally break performance in another.


## The intuition

Imagine a mechanic who fixes a car's engine. Before he hands it back, he does not just drive it on a bumpy road; he puts it on a dynamometer—a machine that measures horsepower under controlled conditions. He runs the old engine specs through the test, then the new ones. If the numbers go up, the fix worked. If they go down, he found a problem. An evaluation dataset is that dynamometer for an AI model.

Here is how it works in practice:
1. You start with 100 specific customer complaints and their ideal responses. These 100 pairs never change during your testing period.
2. You feed the old model those 100 complaints. It gets 85 correct.
3. You update the model to fix a bug in handling angry tones.
4. You feed the *exact same* 100 complaints to the new model. It now gets 92 correct.

The dataset is your measuring stick. It does not need to represent every possible human conversation forever. It just needs to be stable today and tomorrow so you can see the difference. If you added new questions halfway through, you could no longer tell if the score went up because the model improved or because the new questions were easier.

Consider grouping your tests by topic, which is simply taking subsets of your tests based on specific criteria. For a travel bot, you might have 20 questions about flights, 20 about hotels, and 20 about car rentals. If you update the hotel pricing logic, you check the 'hotel group' specifically to ensure those answers are better, while verifying that the 'flight group' didn't get worse. Without these groups, a global score might hide a collapse in just one area.

You also need a rule that prevents releasing updates if performance drops. Think of this as a gatekeeper: before any new software version goes live, it must score at least as well on the test as the previous version did. If it scores lower, the update is stopped. This catches unintended quality drops caused by changes meant for other problems.

You typically run these tests in a controlled environment, separate from real users, to avoid disturbing customers and to allow for careful analysis. This is often called offline testing because it happens away from the live system. Comparing your model against competitors using a shared set of public questions ensures that everyone is judged by the same standard, which allows for fair benchmarking.

For some tasks, like checking if code runs or if dates are formatted correctly, you can use automated checks that give a simple pass or fail result, removing human guesswork. This is deterministic validation because the outcome is always the same for the same input.

Be careful not to let the model memorize the test questions instead of learning the underlying rules. If it does, it scores 100% on your tests but fails with real users who ask new questions. This is called overfitting. Similarly, ensure your test data has not leaked into the training material used to build the model. If the model saw the test answers during its learning phase, its high score is fake; this is known as contamination.


## An analogy

Think of a runner training for a marathon. Every week, they run the same 5-kilometer loop on the exact same path, at the same time of day, in the same shoes. This repeated measurement is their 'evaluation dataset.'

If the runner changes their diet (updates the model), they can look at their times from last month. If the time improves, the diet worked. If the time gets worse, something is wrong.

Now imagine the runner changes the route every week—some weeks on flat pavement, some on hills, some in rain. When they finish this week's run in a great time, they don't know if it’s because their diet improved or because they happened to get an easy downhill day. That chaos is what happens when evaluation sets aren't stable.

To make the analogy useful, the runner also tracks splits: pace per mile. This is like your 'slice-based set.' Maybe the runner is getting faster on flat ground but slower on hills. The overall time might be good, but the split data reveals a weakness to address.

Where this analogy breaks down: a runner’s fitness is continuous and internal. An AI model’s 'fitness' depends on massive amounts of data it was trained on previously, which we cannot easily see or control during the test.


## Definition

An evaluation dataset is a permanent, version-controlled collection of example inputs and their correct outputs, used as a fixed standard to measure whether changes to a model improve or degrade its performance.

Key concepts in plain words:
- Testing before release: Evaluating the model on this fixed dataset in a controlled environment before letting real users interact with it.
- Fair comparison: Comparing your model against others using a common public dataset, ensuring everyone is measured by the same yardstick.
- Automated checks: Using automated rules that give a simple Pass/Fail result for specific tasks, removing human guesswork.
- Memory traps: When a model memorizes test questions instead of learning general rules, leading to failure on new questions.
- Data leakage: When the test data accidentally becomes part of the training material, making high scores meaningless.


## How it connects

This concept is the foundation for offline testing (testing in a lab) and benchmarking (comparing against public standards). It directly prevents the risk of overfitting by requiring held-out data that the model hasn't seen during training. It also enables automated validation checks, which rely on fixed datasets to run consistent tests.


## The mathematics

Evaluation datasets measure how well a model performs on data it has not seen during training. The core mathematics is simple arithmetic: counting, division, and percentages. You do not need algebra, calculus, or vectors.

The primary metric is accuracy. Accuracy is the fraction of correct predictions out of all predictions made. To calculate it, you count how many times the model’s output matched the expected answer (the ground truth), and divide that number by the total number of items in the dataset.

accuracy = (number of correct outputs) / (total number of inputs)

The result is a number between 0 and 1. A score of 1 means every prediction was correct. A score of 0.5 means half were correct.

When comparing two versions of a model, you often want to know the relative improvement. This uses percentage change. You take the new accuracy, subtract the old accuracy, divide by the old accuracy, and multiply by 100 to get a percentage.

percentage change = ((new_accuracy - old_accuracy) / old_accuracy) * 100

This tells you how much better (or worse) the new model is compared to the old one, relative to the old performance.

Another key concept is sampling. In practice, you cannot label every possible input. You must pick a subset of data to evaluate. This subset is called the test set. The test set must represent the whole population of real-world inputs. If common questions make up 90% of real traffic, your test set should also have roughly 90% common questions. This is called stratified sampling. If you pick randomly without regard to frequency, rare errors might be missed or common errors overrepresented.

The goal of using a separate test set is to estimate generalization error. Generalization error is the expected prediction error on unseen data. By measuring performance on the test set, we get an estimate of how the model will perform in the real world, not just on the training data it memorized.


## A worked example

Suppose you have a dataset of 10 questions. For each question, the model either answers correctly (a pass) or incorrectly (a fail). You run Version 1 and Version 2.

Step 1: Calculate accuracy for Version 1.
Version 1 passes 7 out of 10 questions.
The number of correct outputs is 7.
The total number of inputs is 10.
Accuracy for Version 1 = 7 divided by 10 = 0.7.

Step 2: Calculate accuracy for Version 2.
Version 2 passes 8 out of 10 questions.
The number of correct outputs is 8.
The total number of inputs is 10.
Accuracy for Version 2 = 8 divided by 10 = 0.8.

Step 3: Calculate the absolute difference in accuracy.
Subtract the old accuracy from the new accuracy.
0.8 minus 0.7 equals 0.1.
This means Version 2 got 0.1 more correct per question on average.

Step 4: Calculate the percentage change (relative improvement).
Take the difference (0.1) and divide it by the old accuracy (0.7).
0.1 divided by 0.7 equals approximately 0.142857.
Multiply by 100 to convert to a percentage.
0.142857 times 100 equals 14.2857 percent.
Round to one decimal place: 14.3%.

Result: Version 2 is 14.3% better than Version 1 on this dataset.

Note: This calculation assumes the test set is representative of real-world data. If the 10 questions were not a random or stratified sample of the true problem space, this accuracy score may not reflect real performance.


## From scratch, in Python

```python
# No frameworks here. We load a dataset from a plain list.

# A list in Python is an ordered collection of things, in square brackets.
# Here, each item is a small dictionary: a 'question' and the expected 'answer'.
dataset = [
    {"question": "What is 2+2?", "expected": "4"},
    {"question": "Capital of France?", "expected": "Paris"},
    {"question": "What is the colour of the sky?", "expected": "blue"},
    {"question": "Who wrote Hamlet?", "expected": "Shakespeare"},
]

# A function is a reusable block of code. This one takes the dataset and a
# list of model outputs, and tells us how many matched.
def accuracy(dataset, outputs):
    # We assume outputs has the same length as dataset.
    correct = 0
    # A for loop visits each item in order.
    for i in range(len(dataset)):
        if outputs[i] == dataset[i]["expected"]:
            correct = correct + 1
    return correct / len(dataset)

# Example model outputs (imagine they come from running the model).
outputs_v1 = ["4", "Paris", "green", "Shakespeare"]  # 3 correct
outputs_v2 = ["4", "Paris", "blue", "Shakespeare"]   # 4 correct

print(accuracy(dataset, outputs_v1))  # 0.75
print(accuracy(dataset, outputs_v2))  # 1.0

# Now split the dataset into slices: a list of lists, or a dictionary.
# A dictionary is a lookup table by name. A set holds unique items.
common_questions = ["Capital of France?"]
hard_questions = ["Who wrote Hamlet?"]
# A set of all questions, just to check we have no duplicates.
all_questions = {q["question"] for q in dataset}
print(all_questions)
```


## What the code does

We start with a list of dictionaries. A dictionary holds key-value pairs — 'question' maps to the text, 'expected' maps to the correct answer. The for loop goes through each index i from 0 to 3. For each, it compares the model's output (outputs[i]) to the 'expected' value. If they match, we add one. After the loop, correct is the number of matches. We divide by the total number of questions to get a fraction. The example shows two versions of outputs — v1 gets 3 correct, v2 gets all 4. In practice, you would not type outputs by hand; you would run the model on the dataset. The set comprehension {q["question"] for q in dataset} is shorthand for 'build a set of all the question texts' — sets hold each thing once, so it would reveal if you accidentally had duplicate questions.


## Common misconceptions

**A dataset must be huge to be useful** — If I only have 50 examples, my evaluation is worthless. I need at least 10,000.

A small but well-curated dataset that covers the behaviours you care about is far more useful than a huge one that covers trivia. Even 10 questions can catch a regression if they target critical paths. The dataset's value lies in being representative and stable, not in size. Many production evals run on a few hundred hand-picked cases. In fact: A small, targeted dataset that covers your key behaviours is enough to start. You can grow it over time with real traffic samples.

**The whole dataset must be labelled with expected outputs** — Every example needs a gold-standard answer, otherwise it is useless for evaluation.

Some evaluation datasets are unlabelled — they just specify properties. For example, a dataset of prompts for a safety eval might have no correct answer, only a rule: 'the response must not contain personal information'. You can also use model-based judges that score on a rubric, not against a fixed answer. The dataset is a set of inputs with constraints, not necessarily outputs. In fact: Evaluation datasets can be labelled (with expected outputs) or property-based (with expected characteristics). Both are valid.

**Once a dataset is created, it never changes** — I made a dataset last year, it is my sacred test set, and I will use it forever.

Real production traffic drifts. A dataset that was realistic a year ago may no longer represent your users. If you never update, you will get excellent scores while the model fails in production. Evaluation datasets must evolve with the product, and changes must be versioned explicitly so you can still compare old results. In fact: Update your dataset as your product and users change. Version it (v1, v2) and keep the old versions around for historical comparison.

**Accuracy is the only metric that matters** — The accuracy number on my dataset tells me whether my model is good.

Accuracy treats every mistake equally. For fraud detection, missing a fraud case is far worse than wrongly flagging a normal one. On a dataset with 99% non-fraud, a model that says 'not fraud' to everything gets 99% accuracy and is useless. You need different metrics like precision and recall, which weight errors differently. In fact: Accuracy is only one metric. Depending on the task, choose metrics that reflect the relative cost of different errors.

**Real production traffic is not needed for evaluation** — I can build a perfect dataset from my imagination or from public benchmarks, and it will tell me how my model will behave in production.

Public benchmarks are contaminated by training data and rarely match your task distribution. Imagined examples miss the weird, messy inputs real users produce — misspellings, niche jargon, ambiguous phrasing. Real production traffic is the best source of realistic cases, because it captures what your users actually ask. In fact: Build your evaluation datasets from real traffic, supplemented by synthetic cases. Public benchmarks are a starting point, not the final word.


## In short

An evaluation dataset is a fixed, curated, versioned set of inputs with expected outputs or properties. It is the measuring stick that makes comparisons possible. Keep it stable for the duration of any comparison, evolve it with your product, always version changes, and draw from real traffic. A small, representative dataset beats a huge random one. This is the foundation of every evaluation you will ever run.


## Where this is used

In large language models, a training run on billions of tokens uses evaluation datasets to decide when to stop. Developers hold out a small, fixed set of prompts — say, 1000 English grammar checks — and after each training step they run the model on it. If the score stops improving, they stop training. Models like GPT-4 are compared on public benchmarks like MMLU, which is an evaluation dataset. The key mechanism: the dataset never changes during the entire training run, so the score trend is trustworthy. In small models running on devices (TinyML), evaluation datasets are tiny — often just a few hundred keywords or sensor patterns — but they must be collected from the actual device's microphone or accelerometer, because the distribution differs from lab data. For retrieval-augmented generation (RAG), you need a dataset of questions and the factual answer plus the passage that supports it. The evaluation dataset includes not just the expected answer but the expected retrieved context. Developers build a 'golden set' where each question has a specific chunk of text that should be retrieved. A RAG system is evaluated by whether the right chunk is retrieved and whether the answer is correct. In agentic systems that plan and call tools, evaluation datasets are the hardest. A dataset entry includes a user request, the expected sequence of tool calls, the arguments to each call, and the final answer. For example, {'request': 'book a flight to Delhi tomorrow', 'expected_tool_calls': [{'tool': 'search_flights', 'args': {'from': 'Mumbai', 'to': 'Delhi', 'date': 'tomorrow'}}]}. The dataset is used to check that the agent does not skip steps or call tools in the wrong order. This is 'gold set' territory: you are specifying exact expected behaviour. In everyday applied work, evaluation datasets are ubiquitous. For a fraud-detection model, the dataset contains past transactions labelled as 'fraud' or 'not fraud'. For a recommendation system, it contains user-item pairs with a rating or a click label. The dataset is split into train and test. Developers run the model on the test set after training to measure accuracy, precision, recall — all computed from the fixed dataset. The critical mechanism is that the test set is never touched during training, only at the end. The book 'Practical MLflow for Generative AI on Databricks' emphasises the same: evaluation is what turns 'I like version 2 better' into 'version 2 is measurably better on the behaviours we care about'. In practice, a developer writing an evaluation pipeline will use a library like MLflow or a custom Python script. With MLflow, they write: mlflow.evaluate(data=eval_dataset, model=model) — but underneath, MLflow is doing exactly the same counting loop: compare each prediction to the expected value, accumulate correct, divide by total. The library does not change the concept.


---

Read this on the web: https://basavapurushottam.com/learn-ai/evaluation-datasets
