# Synthetic Eval Data

Generating test cases with models to reach coverage that manual authoring cannot, with careful validation.

**Rests on:** Evaluation Datasets

*LLMs and Generative AI · LLM Evaluation*


## Why learn this

You need this because human-written test cases alone cannot catch every failure your AI model will have in the real world. Synthetic eval data lets you generate thousands of test cases automatically, reaching corners that manual authoring would never cover. Without it, you will ship a chatbot that fails on the one phrasing your users actually use, or a fraud detector that misses a rare but costly pattern. Master this and you will be able to build evaluation sets that actually measure whether your model works, which is the first step toward safe deployment, monitoring, and improvement. This unlocks further study in LLM Evaluation, AI Safety, and MLOps.


## The intuition

Synthetic eval data means using a model to create test cases for another model. Think of it as a teacher writing practice questions for a student, except the teacher is also a computer program. The purpose is to get more coverage than a human could write by hand. A person might write twenty test questions; a generator can write twenty thousand, each slightly different. But there is a catch: the generator has its own blind spots. If the generator never saw a certain kind of question, it will never create that kind of test case. So synthetic data is not a replacement for human effort; it is a tool to expand the range of what you test, while humans review a sample to keep it honest.


## An analogy

Imagine you are a coach training a cricket team. You cannot play every possible opponent, but you can simulate different bowling styles with a bowling machine. The machine can send balls at different speeds, angles, and spins — far more than any human bowler could on a given day. That is synthetic eval data. You set the machine to produce variety, and the players practice against it. But the machine has limitations: it cannot recreate a bowler's subtle change in grip, or the psychological pressure of a live match. So you still need real matches against real teams to see how the players perform under genuine conditions. The machine gives you volume and coverage; real matches give you authenticity. Similarly, synthetic eval data gives you volume and coverage, but human review and real-world test cases keep it grounded. The analogy breaks down at one point: a bowling machine still sends balls physically, but synthetic data is generated by a model that itself might have biases. The machine is not neutral; it is a tool that reflects the data it was trained on.


## Definition

Synthetic eval data is test cases generated by a model, rather than written by a human, used to evaluate another model's performance, with the understanding that human review of a sample is necessary to catch the generator's blind spots.


## How it connects

You have already studied LLM Evaluation as a decision system, and Evaluation Datasets as the fixed reference for comparisons. Synthetic eval data is a way to build those datasets faster and more broadly. It also connects to Gold and Adversarial Sets: you can generate both kinds synthetically. And it relates to Evaluator Drift, because if your generator model changes, your eval data set changes, which invalidates comparisons — exactly the drift problem you have notes on.


## The mathematics

The mathematics behind synthetic evaluation data is not about complex equations; it is about counting, probability, and estimating how well a small sample represents a large whole.

Imagine you are testing a chatbot that handles customer queries. The "input space" is the set of all possible ways a human could ask a question. This space is too large to check one by one. For example, asking "What time does the train leave?" and "When does the next train depart?" are different inputs but belong to the same category: checking departure times.

Because you cannot test every possible phrasing, you generate a sample of test cases using a synthetic generator. The generator draws these cases from a probability distribution over the input space. This means each generated case is a random pick from the universe of possible questions. The more cases you generate, the more likely you are to cover diverse situations. However, you can never be certain you have covered everything. Instead, you estimate coverage and quality.

To manage this, we use two main mathematical ideas: stratification and sampling estimation.

Stratification ensures variety. If you know your input space has distinct categories (such as 'booking', 'cancellation', or 'delay'), you do not just pick randomly from the whole space. You divide the space into these categories, called strata. You then decide how many test cases to generate for each category. This is simple multiplication:

Total Test Cases = (Number of Categories) × (Test Cases per Category)

For example, if you have 3 categories and want 100 tests for each, you generate 300 cases. This guarantees that every category is represented in your test set.

Estimation ensures quality. You cannot check all 300 cases by hand. So, you take a random sample from the generated set. A random sample means every case has an equal chance of being picked, just like drawing names from a hat. You then manually verify each case in this sample to see if it is valid (makes sense) or invalid (nonsense).

Let:
- n be the size of your manual review sample (the number of cases you check).
- k be the number of those n cases that are valid.

The estimated validity rate of the entire set is the ratio of valid cases in your sample to the total size of your sample:

Estimated Validity Rate = k divided by n

If you check 10 cases (n=10) and 8 are valid (k=8), the estimated validity rate is 8/10, or 0.8 (80%). You can then estimate the total number of valid cases in your full set of 300 by multiplying the total size by this rate:

Estimated Valid Cases = Total Test Cases × Estimated Validity Rate

In our example: 300 × 0.8 = 240 valid cases.

This is basic statistics. It relies on the law of large numbers, which states that if your sample is random and large enough, the proportion of valid items in the sample will closely match the proportion of valid items in the whole population. We do not need algebra to understand this; we only need to count and divide.

Note: This approach assumes the generator produces independent samples from a defined distribution. If the generator is biased (e.g., it always generates 'booking' questions), the random sampling will not give an accurate estimate of the whole space's quality, because the sample does not represent the true diversity of inputs.


## A worked example

Imagine you are building test cases for a chatbot that answers questions about train schedules. You have identified three categories: 'booking', 'cancellation', and 'delay'. You want to generate 100 test cases per category using a synthetic generator.

Step 1: Calculate the total number of generated test cases.
You have 3 categories. You decide to generate 100 test cases for each category.
Calculation: 3 × 100 = 300.
Result: You now have a pool of 300 synthetic test cases.

Step 2: Select a random sample for manual review.
You cannot check all 300 cases by hand. You choose to check a small subset. Let us say you pick 10 cases at random from the 300. This is your sample size, n.
n = 10.

Step 3: Manually verify the sample.
You read each of the 10 cases and decide if it is valid (a sensible question) or invalid (nonsense). Suppose you find that 8 of them are valid and 2 are nonsense.
k = 8 (the number of valid cases in your sample).

Step 4: Calculate the estimated validity rate.
You divide the number of valid cases by the sample size.
Calculation: 8 divided by 10 = 0.8.
Result: The estimated validity rate is 0.8, or 80%.

Step 5: Estimate the total number of valid cases in the full set.
You apply this rate to the total pool of 300 cases.
Calculation: 300 × 0.8 = 240.
Result: You estimate that approximately 240 of your 300 generated test cases are valid.

Step 6: Check for coverage (variety).
You look at the 10 cases in your sample to ensure they represent all categories. Suppose you see:
- 4 'booking' questions
- 3 'cancellation' questions
- 3 'delay' questions
This shows that your random sample captured variety across all three categories, giving you confidence that the full set of 300 is also diverse.

Final Outcome: You have 300 test cases. Based on your sample, you estimate that 240 are usable. You keep those 240 and may regenerate or discard the rest depending on your quality thresholds.


## From scratch, in Python

We will write a small Python script that simulates generating synthetic test cases. We will not use any AI library here; we will just show the logic of sampling and validation. First, we need a list of categories and a simple way to generate a random test case per category. We will use the `random` module, which is part of Python's standard library.

```python
import random

# A list holds categories in order: booking, cancellation, delay
categories = ['booking', 'cancellation', 'delay']

# We want 100 test cases per category
cases_per_category = 100

generated_cases = []  # an empty list to hold all generated test cases

# For each category, loop 100 times using a for loop
for category in categories:
    for i in range(cases_per_category):
        # Create a test case as a tuple: (category, question_text)
        # We randomly pick a question template from a small set
        templates = [
            "How do I {action} my ticket?",
            "I want to {action} a reservation.",
            "What is the process to {action}?"
        ]
        # The action depends on the category
        if category == 'booking':
            action = 'book'
        elif category == 'cancellation':
            action = 'cancel'
        else:  # delay
            action = 'check a delay'
        template = random.choice(templates)
        # Replace the {action} placeholder with the action word
        question = template.format(action=action)
        # Add the tuple to the list
        generated_cases.append((category, question))

# Now we have 300 test cases. Let's sample 10 for review.
# We pick indices using random.sample, which chooses without replacement
sample_indices = random.sample(range(len(generated_cases)), 10)
sample = [generated_cases[i] for i in sample_indices]

# Manually review: this is where a human would look. We simulate by checking if the question contains the action word.
valid_count = 0
for case in sample:
    category, question = case
    if action in question:  # actually we need to check each case correctly; let's fix that later
        valid_count += 1

# But note: in the loop above, 'action' variable is from the last iteration of the earlier loop, which is wrong. Let's rewrite properly.

# Correct approach:
valid_count = 0
for case in sample:
    category, question = case
    # The expected action for this category
    if category == 'booking':
        expected = 'book'
    elif category == 'cancellation':
        expected = 'cancel'
    else:
        expected = 'check a delay'
    if expected in question:
        valid_count += 1

# Now compute validity estimate
validity_estimate = valid_count / len(sample)
print("Generated", len(generated_cases), "cases.")
print("Sample validity estimate:", validity_estimate)

# Check coverage across categories
category_counts = {c: 0 for c in categories}
for case in sample:
    category, question = case
    category_counts[category] += 1
print("Sample category counts:", category_counts)
```


## The same thing, with a framework

In practice, you would use an LLM API to generate the test cases. For example, you might use the OpenAI Python client to send a prompt and get a response. Here is a sketch (not runnable without an API key):

```python
import openai

client = openai.OpenAI(api_key='your_key')

categories = ['booking', 'cancellation', 'delay']
for category in categories:
    prompt = f"Generate 100 diverse user questions about train {category} (e.g., booking tickets, cancelling, delays). Output one question per line."
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[{"role":"user", "content":prompt}],
        temperature=0.7
    )
    questions = response.choices[0].message.content.strip().splitlines()
    # then add them to your eval dataset
```

Under the hood, the API call does what your loop did: it generates text based on a prompt. The difference is that the model has a much larger and more varied set of templates than you wrote.


## What the code does

The code generates a list of 300 test cases by looping over each category and creating 100 examples. Each test case is a tuple: (category, question). We use `random.choice` to pick a template, and `format` to fill in the action word. After generation, we sample 10 cases using `random.sample`, which picks unique indices. We then manually check each sample: we verify that the question contains the expected action for that category. That is a simple heuristic substitute for human review. Finally, we compute the validity estimate and count how many sample cases fall in each category. This shows the basic workflow: generate, sample, validate, estimate. In real life, the generator would be an LLM, and the validation would be a human reading the questions, not a string check.


## Common misconceptions

**Synthetic data is as good as human data** — If I generate 10,000 synthetic test cases, I don't need to manually write any.

Synthetic data inherits the generator's blind spots. If the generator never produces a certain kind of input (e.g., a question in Hindi or with a typo), your eval set won't have it. In fact: Synthetic data is a complement to human-written data, not a replacement. You need a human-reviewed core set, and synthetic data broadens coverage around it.

**The more synthetic data, the better** — I can just generate a million test cases and my eval will be perfect.

Quantity does not guarantee quality. If the generator is biased, more cases just repeat that bias. Also, you have to review a sample to estimate validity, and if the validity is low, you waste effort. In fact: You need to validate a sample and adjust the generator based on that. More data helps only if it is diverse and valid.

**The generator is neutral** — Because a model generates the data, it is objective and free of human bias.

Models are trained on human data and replicate human biases. The generator will produce test cases that reflect its training distribution, which may not match your target population. In fact: Treat the generator as a biased source. Review its output, compare it to real user data, and adjust your prompts to steer it toward the diversity you need.


## In short

Synthetic eval data is generated test cases that expand coverage but need human validation. Use it to complement, not replace, human-authored test sets. Always sample and review to catch the generator's blind spots.


## Where this is used

Synthetic eval data is used across AI systems to build evaluation sets that are broad and varied. In large language models, developers generate hundreds of thousands of test prompts covering different phrasings, domains, and edge cases. For example, to test a summarisation model, they create synthetic news articles, conversations, and legal documents. In small models running on devices (like a phone or a smart speaker), synthetic data is used to test keyword spotting across different accents, background noises, and languages, because recording real audio for every scenario is impossible. In retrieval-augmented generation (RAG), synthetic eval data is used to generate queries that test retrieval — you create synthetic questions and then check whether the right document is retrieved. In agentic systems that plan and call tools, you generate synthetic user goals and tool schemas to test whether the agent picks the right tool and handles errors — you can simulate hundreds of tool calls. In everyday applied ML (recommendation, fraud detection, forecasting), synthetic data is used to test the model on rare but important events, such as fraudulent transactions that occur 0.01% of the time. A developer writes a script that samples from a distribution of features and generates test cases, then uses a model validator or a human to check them. The library is usually an LLM API, and the developer writes a prompt that specifies the diversity and constraints. What breaks when done badly? If the generator has a blind spot, your eval set will miss that failure, and you will ship a bug. For example, if you generate synthetic test cases for a chatbot but never include slang or regional dialects, the chatbot will fail on those in production. So human review of a sample is the safeguard.


---

Read this on the web: https://basavapurushottam.com/learn-ai/synthetic-eval-data
