# Evals Gap

The widespread pattern of shipping LLM applications with no real measurement discipline, so quality regressions go unnoticed.

**Rests on:** LLM as Judge

*LLMs and Generative AI · LLM Evaluation*


## Why learn this

This concept explains why some teams' AI tools work reliably in daily use while others break silently when conditions change slightly. Mastering it gives you a practical skill: you can look at any artificial intelligence project and see if it has a routine way to check for quality problems, and if it does not, you can close that gap with a small, honest set of test cases. This is the foundation for all other measurement techniques — such as building fixed reference sets, using simple yes-or-no checks, distinguishing between relevant and correct answers, comparing text outputs automatically, measuring performance against standards, noticing when your own tests become unreliable, testing both easy and hard questions, and balancing quality against safety and cost — because every one of those builds on the simple act of deciding what to measure, recording it, and checking it on every change.


## The intuition

Imagine you are a chef who has just launched a new dish at your restaurant. You taste it once before you serve it, it seems fine, and you put it on the menu. You do not taste it again for a month. Diners start complaining — the sauce is too salty, the chicken is dry — but you have no idea when it changed, which ingredient caused it, or whether the new oven is the culprit. You only know that people are unhappy. Now imagine a chef who keeps a notebook. Every day, they cook the same sample portion, taste it, rate it on a scale of one to ten for saltiness, dryness, and flavour, and write the number in the notebook. If the rating drops, they know exactly when it dropped, and they can trace back to what changed that day — the new salt brand, the new cooking oil, the new recipe tweak. The second chef has a measurement discipline. The first has an evals gap. The evals gap, in AI, is the distance between a team that ships an artificial intelligence application with no systematic way of checking whether it still works, and a team that can answer, with evidence, 'Is this version better or worse than the last, and by how much?'


## An analogy

Think of a bridge. A bridge is designed, built, and opened to traffic. If nobody ever inspects it — no load tests, no crack monitoring, no routine checks — then a small crack in a support beam goes unnoticed for months, until one day the bridge fails catastrophically. The cost of failure is enormous, and it could have been caught with a small, cheap inspection regime. The evals gap is the same thing for AI systems. An LLM application is a bridge: it is built on a foundation (a model), carries traffic (users), and is subject to stress (new prompts, new data, unusual edge cases). Without inspection — without a fixed set of test prompts, a way to score the outputs, and a habit of running that score on every change — a regression slips through. The demo of the bridge, at opening day, proves nothing about its state six months later. And the demo of an LLM app — 'look, it answered this question beautifully' — proves nothing about how it will behave with a user who phrases things differently, or asks something slightly out of scope. Where the analogy breaks down: unlike a physical bridge, an AI system's behavior can shift unpredictably due to updates in its core model or subtle changes in your own code, meaning you must run these checks frequently and track every version carefully.


## Definition

The evals gap is the widespread failure to build any systematic measurement into artificial intelligence applications — meaning that quality regressions go unnoticed until users complain, because no one has defined what 'working' looks like in measurable terms, collected a stable set of inputs and expected outcomes, and checked every change against that set.


## How it connects

You have not yet covered any other evaluation concepts, so this page stands alone as your introduction. But it connects forward to everything listed under LLM Evaluation: Evaluation Datasets (the fixed reference set you'll build), Deterministic Validators (the simplest checks that can run first), Answer Relevancy and Correctness (the distinction between whether an answer addresses the question and whether it's true — both need measurement), BLEU and Reference Metrics (automatic ways to compare text), Benchmarking (comparing models on standard tasks), Evaluator Drift (the risk that your measurement itself changes and becomes unreliable), Gold and Adversarial Sets (the two kinds of test data you'll want), and the Four-Axis framework (the reminder that quality is only one axis — safety, cost, and reliability also need measurement). The evals gap is the problem; all those concepts are pieces of the solution. You are starting at the beginning, which is exactly right.


## From scratch, in Python

```python
# No frameworks, just plain Python. We will build a tiny eval harness.
#
# First, the building blocks we need:
# - a list: an ordered collection that can be changed. e.g. ["hi", "hello", "bye"]
# - a dictionary: looks things up by name. e.g. {"good": 3, "bad": 0}
# - a for loop: repeats a block for each item in a list.
# - a function: a named block of code we can reuse.

# --- Step 1: Define a tiny evaluation dataset.
# Each entry is a dictionary: {"input": ..., "expected": ...}
# The 'expected' is the minimum length of a good answer, say.

eval_set = [
    {"input": "What is the capital of France?", "expected_min_len": 5},
    {"input": "Explain photosynthesis in one sentence.", "expected_min_len": 10},
    {"input": "List two programming languages.", "expected_min_len": 5},
]

# --- Step 2: Define the 'system under test'.
# In real life this would be your LLM call, but here we fake it.
def fake_model(prompt):
    # A pretend model that returns a fixed, short answer.
    return "Short."

# --- Step 3: A function to evaluate the model on our set.
def evaluate(model, dataset):
    results = []  # We will collect per-item results.
    for item in dataset:
        output = model(item["input"])
        passed = len(output) >= item["expected_min_len"]
        results.append(passed)  # True or False
    return results

# --- Step 4: Run it and compute the pass rate.
results = evaluate(fake_model, eval_set)
pass_count = 0
for r in results:
    if r:
        pass_count = pass_count + 1
pass_rate = pass_count / len(eval_set) * 100
print("Passed", pass_count, "out of", len(eval_set), "=>", pass_rate, "%")

# --- Step 5: Imagine a better model that actually answers.
def better_model(prompt):
    if "capital" in prompt:
        return "Paris."
    elif "photosynthesis" in prompt:
        return "Plants make food from sunlight."
    else:
        return "Python and Java."

results_better = evaluate(better_model, eval_set)
pass_count_better = sum(1 for r in results_better if r)
print("Better model pass rate:", pass_count_better / len(eval_set) * 100, "%")

# This is the seed of an eval harness: a fixed dataset, a scoring rule,
# and a function that runs the model and reports pass rate.
```


## The same thing, with a framework

```python
While there is no single standard library yet, tools like Promptfoo and LightEval wrap this exact loop. For example, in Promptfoo you would define a YAML config with test cases and assertions, and run `promptfoo eval`. Underneath, it does the same thing as our Python loop: iterates over cases, calls the model, checks assertions, and reports pass rates. The library is not magic; it is a config-driven version of the code we just wrote.
```


## What the code does

Walk through the plain Python: We define `eval_set` as a list of dictionaries. Each dictionary has an `input` (the prompt) and an `expected_min_len` (the minimum length a good answer should have — a crude but real measurement). Then `fake_model` is our stand-in for an LLM; in a real project it would be an API call. The `evaluate` function takes a model and a dataset. It loops over each item, calls the model, checks the condition (length >= minimum), and stores True or False in a list. Finally, we count the Trues and compute a pass rate. Then we define a slightly better model and see it pass more. The point: this tiny loop is the core of any evaluation system — a fixed set of inputs, a scoring rule, and a repeated run. All frameworks (Promptfoo, LightEval, etc.) are just this, scaled up with more sophisticated scoring and reporting.


## Common misconceptions

**The demo tells you it works** — I showed it to my friend and it answered perfectly. So it's ready to ship.

A demo is a single point in the input space. It proves nothing about how it will behave on inputs you haven't tried. The set of all possible user inputs is astronomically large; a demo samples one point from it. In fact: A demo is a necessary sanity check, not an evaluation. Evaluation requires a systematic sample of many inputs, with a scoring rule, compared over time.

**An eval set is a nice-to-have, not essential** — My app is simple. I don't need all this eval stuff.

Simple apps still have input variation. A single user might phrase a question in a way that triggers a wrong answer. Without a fixed set of test cases, you cannot detect that the new prompt tweak you made broke that case. In fact: Even a 10-item eval set — covering the most common cases and a few edge cases — is vastly better than none. You can start small and grow it.

**Evals are only for model training, not for production** — The model was already evaluated by the company that made it. I just use the API.

The model's own benchmarks are about the model in isolation. Your app adds your own prompts, your own context, your own logic (like RAG retrieval). The model's performance doesn't tell you how your *app* performs. In fact: You must evaluate the whole system (model + your code) on your own data. The model's benchmarks are a starting point, not a finish line.

**One test case is enough** — I checked that it handles the common case. That's all anyone will do.

LLMs are stochastic and can fail on small variations. A single test case gives you no confidence about the distribution of outcomes. One in a hundred failures is still a failure. In fact: You need a *set* of test cases, ideally covering different categories (gold, adversarial, slices). The more varied the set, the more it represents real usage.

**A higher score always means better quality** — My eval score went up, so my app is better.

The eval set defines what 'better' means. If the set is unrepresentative (all easy questions), a higher score might mean it got better at easy questions but worse at hard ones. Also, a metric can be gamed — e.g., if you score by length, the model can get a higher score by being more verbose, not more correct. In fact: Your eval set must be a representative sample of what your users will actually ask. And you need to decide what the metric really measures — length isn't correctness. You also need to consider the four axes: quality, safety, cost, and reliability.

**If users aren't complaining, it's working** — Nobody has complained, so the app is fine.

Most users don't complain; they just quietly stop using the app, or they don't return. Complaints are a lagging indicator — by the time you hear about a problem, many users have already had the bad experience. In fact: You need a proactive measurement loop. Run your eval set regularly, even if nothing is changing, to catch drift before users notice.

**Evaluating is too expensive and slow** — Running an eval set costs money and takes time. Shipping fast is more important.

The cost of an eval run is often pennies and minutes, compared to the cost of a broken deployment that damages user trust and requires engineering time to debug. Also, many checks can be deterministic and cheap (format, length). In fact: Start small: a handful of test cases, scored deterministically (e.g., regex, length) first, then add more expensive checks (LLM judge) as needed. The cost is an investment in knowing whether your app still works.


## In short

The evals gap is the mistake of shipping an LLM application without a systematic way to measure whether it still works. A demo is not an evaluation. The fix is simple and cheap: build a small, honest eval set — a list of inputs and expected outcomes — and run it on every change. Close the gap with a regression gate that blocks changes that drop your pass rate. Evaluation is a decision system, not a score; if the metric gates nothing, it's decoration.


## Where this is used

The evals gap shows up everywhere in AI work. **In large language models (LLMs)**: a model is trained and then, before release, evaluated on a benchmark suite. But the gap is in the *deployment* stage: when you, the developer, take that model and build an app, you must create your own eval set tailored to your app's task. If you don't, you are flying blind — the model's pre-training benchmarks don't tell you how it will answer your specific users' prompts. For example, if you are building a customer-support bot, you need a set of typical customer questions, and you need to score the model's answers for correctness and helpfulness on those questions. **In small models on devices (TinyML, on-device LLMs)**: the constraints are even tighter. The model runs on a phone or edge device, where memory and latency matter. The eval gap here means: a model that appears accurate on a server can fail on a device because of quantization (reducing precision to save memory), which slightly changes outputs. So you must evaluate the *deployed* version, not just the original. **In retrieval-augmented generation (RAG)**: the system combines a retrieval step (fetching relevant documents) and a generation step (writing an answer using those documents). The eval gap is everywhere: if the retriever fetches irrelevant docs, the generator will hallucinate; if the generator ignores the docs, it's no better than a bare model. You need separate evals for retrieval quality (are the right documents found?) and for generation quality (is the answer grounded in the docs?). Without an eval set of queries and ground-truth documents, you cannot tell which component is failing when the final answer is bad. **In agentic systems (planning and tool use)**: agents are more complex because they make multiple calls, call tools, and iterate. An eval gap here is catastrophic — a bug in how the agent formats a tool call can silently break the whole flow. You need eval sets that exercise the full trajectory: a script of what the agent should do at each step. Testing only the final answer misses the failures in the intermediate steps. **In everyday applied machine learning** (recommendation, vision, forecasting, fraud detection): the same principle applies — a model is deployed, and if no one re-evaluates on new data, performance drifts (the world changes, user behaviour changes, fraud patterns change). The evals gap is the missing feedback loop that would catch this. **What a developer actually writes**: they create an eval dataset (like our Python list of dictionaries), they choose scoring metrics (exact match, BLEU, LLM-as-judge, human review), and they set up a test that runs on every code change — a regression gate. If the pass rate drops below a threshold, the change is rejected. They also log evaluation results over time to detect drift. **What breaks when it's done badly**: when there is no eval set, or when the eval set is tiny and unrepresentative, or when the scoring is subjective and unrepeatable, or when the eval set is not versioned (so you change the questions and can't compare old and new scores), or when the judge is an LLM that drifts as it gets updated. The specific failure: you ship a change that makes the app worse for 20% of your users, and you have no idea because the demo looked great. The evals gap is the difference between a team that says 'I think it's fine' and a team that says 'Here is the pass rate on our regression set, it dropped from 98% to 96%, we need to investigate before we ship.'


---

Read this on the web: https://basavapurushottam.com/learn-ai/evals-gap
