# Minimum Evaluable Product

Building the smallest version that can be measured, rather than the smallest that can be demonstrated.

**Rests on:** Evaluation Datasets

*LLMs and Generative AI · LLM Evaluation*


## Why learn this

This concept helps you stop guessing whether your artificial intelligence system works and start knowing. It is the foundation of LLM Evaluation — the practice of checking if a generative or reasoning system actually performs well on quality, safety, cost, and reliability. Without it, you risk building systems that look good in small tests but fail when used by real people. This failure pattern is known as the Evals Gap.

By mastering the Minimum Evaluable Product, you gain the ability to use specific tools effectively:
- Evaluation Datasets (the fixed list of questions or cases you test against);
- Deterministic Validators (automated checks that give a clear yes/no answer); and
- LLM-as-Judge (using one AI to score another).

Ultimately, this concept allows you to set up regression gates — automated checks that block deployment if metrics drop. These gates ensure that new updates do not break what already works.


## The intuition

Here is what a Minimum Evaluable Product (MEP) is not. It is not the smallest feature you can show off to a stakeholder — the kind of thing where you type a question, get a plausible answer, and everyone nods in agreement. That is a Minimum Demonstrable Product. The problem with a demo is that it shows you one thing working well, but never shows you the many other things failing. It creates false confidence.

An MEP is the smallest version of your system that you can actually measure. 'Measure' means: you have a fixed set of test inputs, a defined way of scoring each output, and a number you can compute that tells you whether the system got better or worse after a change.

The test inputs are your Evaluation Dataset. The scoring is done by an evaluation harness — which I will clarify now as the tool or code that runs the tests and calculates the score. This harness might be a deterministic validator, a reference metric like BLEU, or an LLM judge. The number produced by this process is the metric.

Why does this distinction matter? Because when you can measure a thing, you can tell whether a change helped or hurt. When you can only demo a thing, you have anecdotes. The MEP forces you to build the harness first, before you polish the feature. The first milestone shifts from 'impressive' to 'informative' — you learn something real about your system, even if what you learn is that it is bad. That information is worth more than a good demo, because it tells you what to fix.

A smaller MEP is always better, as long as it is measurable. You can start with three test inputs and a rule that checks whether the output contains a required keyword. That is an MEP if it gives you a pass/fail signal you trust. You cannot start with zero test inputs — then you have nothing to measure, and you have slipped back into demo mode.

The key idea, from Practical MLflow for Generative AI on Databricks: 'Ship a Minimum Evaluable Product — the smallest thing that can be measured, not merely demoed.' The book's emphasis is on the word 'measured'. A demo is a story; a measurement is a fact.

Let me give you a concrete shape. Suppose you are building a customer-support chatbot that should answer questions about your return policy. A demo version: you type 'What is your return period?' and it answers '30 days.' All eyes light up. An MEP version: you write down ten questions, including a tricky one like 'Can I return a gift card?' You run the bot on those ten, and you score each answer with a simple rubric: does the answer contain the correct return period? Does it politely refuse when the policy does not apply? Your bot gets 7 out of 10. You learn that it handles the happy path but stumbles on edge cases. That is information your demo could never give you.

That is the whole idea: build the smallest thing that produces a number you can trust, and use that number to guide your next move.


## An analogy

Think of a chef opening a restaurant. The naive chef writes a lavish menu, spends months perfecting a signature dish, and holds a grand tasting for friends. Everyone says it is delicious. But the tasting tells you nothing about whether the restaurant will survive — whether the dish can be cooked consistently during a dinner rush, whether the ingredients stay in budget, whether customers actually order it. The chef has built a Minimum Demonstrable Restaurant: it looks great, but it cannot be evaluated.

The disciplined chef does the opposite. They open a stall with one dish, a notepad, and a simple rule: every customer who finishes the dish and smiles gets marked as a 'yes', everyone who leaves it half-eaten gets a 'no'. After a week they have a number: 8 out of 10 customers finish the dish. They also have notes on why some did not — the sauce was too salty, the portion too small. With that data they adjust the recipe, and the number moves. They have built a Minimum Evaluable Restaurant: the smallest thing that produces a measurement they can act on.

The stall is not the final restaurant. It is the evaluation harness that the real restaurant will later plug into. The notepad and the smile rule are the metric and the scorer. The first milestone is not 'we have a great dish' — it is 'we have a reliable way to know whether the dish is good.'

Where the analogy breaks down: a restaurant's feedback loop takes days, and the smile rule is hopelessly crude. In AI, the feedback loop can be minutes or seconds, and your evaluators — deterministic checks, reference metrics, LLM judges — are far more precise. You can also run your MEP on many more inputs than a chef can serve customers. The analogy understates how fast and how thorough AI evaluation can be, but it captures the core shift: from impressing to informing.


## Definition

A Minimum Evaluable Product (MEP) is the smallest version of a system that can be run against a fixed set of test inputs and scored, producing a measurement that tells you whether the system works and how it changes as you modify it.


## How it connects

You have not yet studied anything in the LLM Evaluation tree, so let me map where this fits. This concept is the root. Everything else grows from it.

Your library lists your planned notes: Answer Relevancy and Correctness (how well the answer matches the question and facts), BLEU and Reference Metrics (automatic ways to compare text against known good answers), Benchmarking (testing against standard industry problems), Deterministic Validators (cheap, unambiguous checks like regex or keyword matching), Evals Gap (the failure of moving from demo to production), Evaluation Datasets (the curated inputs you measure against), Evaluator Drift (when your scoring method itself becomes unreliable over time), Four-Axis Quality Safety Cost Reliability (the main areas to assess), Gold and Adversarial Sets (trusted perfect answers and deliberately tricky traps), and LLM Safety Benchmarks (tests for harmful outputs). When you get to those, notice how each one answers a question that the MEP raises. The MEP asks: 'What is the smallest thing I can measure?' Evaluation Datasets answer: 'Here are the fixed inputs you measure against.' Deterministic Validators answer: 'Here is a cheap, unambiguous way to score some of those inputs.' BLEU answers: 'Here is an automatic way to compare text to a reference.' The Evals Gap is the failure mode this concept exists to prevent. So as you move forward, every note you take will circle back to the MEP's central demand: build the harness first.


## The mathematics

The Minimum Evaluable Product (MEP) is not a complex formula. It is a way of turning a list of results into two simple numbers: a pass rate and an average score. To understand the math, you only need to know how to count, add, divide, and multiply by 100.

The core idea is that you are measuring performance over a fixed set of trials. In this context, a 'trial' is one input given to your system (like a question for a chatbot). The 'evaluation dataset' is the complete list of inputs you have chosen to test. You do not guess or estimate; you calculate exactly what happened on these specific trials.

There are two main calculations you will perform:

1. The Pass Rate
This tells you how often the system succeeded. It is a proportion, which you can also express as a percentage.

To find it, you need two numbers:
- N: The total number of inputs in your evaluation set. This is just the count of items in your list.
- P: The number of those inputs that met your success criteria (the 'passes').

The calculation is:
Pass Rate = P divided by N

If you want a percentage, multiply the result by 100.

2. The Average Score
This tells you how well the system performed on a scale (for example, 1 to 5), rather than just whether it passed or failed.

To find it, you need:
- A list of scores for each input: s₁, s₂, ..., sₙ. Here, 's' stands for score, and the subscript number tells you which input in the list it belongs to. So s₁ is the score for the first input, s₂ for the second, and so on.
- N: The total number of inputs (the same N as above).

The calculation has two steps:
Step A: Add all the scores together. This sum is written as s₁ + s₂ + ... + sₙ. In plain words, this means 'take the first score, add the second, add the third, and keep going until you have added every score in the list.'
Step B: Divide that total sum by N.

Average Score = (s₁ + s₂ + ... + sₙ) divided by N

Why do we divide by N? Because an average is a 'fair share' of the total. If you have a total amount of success spread across N trials, dividing by N tells you how much success each trial contributed on average.

The MEP concept relies on these two numbers because they are stable and comparable. If you change your system and run the same evaluation set again, you can compare the new pass rate or average score to the old one. The math itself does not change; only the numbers P, s₁, etc., will change based on your system's performance.


## A worked example

Let us walk through the calculations step by step with real numbers.

Scenario: You are testing a chatbot that answers questions about a return policy. You have created an evaluation set of 10 specific questions. This means N = 10.

Part 1: Calculating the Pass Rate
Your success rule is simple: the answer must contain the phrase '30 days'.
You run the bot on all 10 questions and record whether it passed or failed.
The results are:
1. pass
2. pass
3. fail
4. pass
5. pass
6. fail
7. fail
8. pass
9. pass
10. fail

Step 1: Count the passes (P).
Look at the list above. Count how many times the word 'pass' appears.
There are 6 'pass' results.
So, P = 6.

Step 2: Identify N.
You tested 10 questions.
So, N = 10.

Step 3: Calculate the pass rate.
Formula: Pass Rate = P divided by N
Calculation: 6 divided by 10 = 0.6

Step 4: Convert to a percentage (optional but common).
Formula: Percentage = Pass Rate multiplied by 100
Calculation: 0.6 × 100 = 60%

Result: The pass rate is 60%. This means the bot gave a correct answer for 6 out of every 10 questions.

Part 2: Calculating the Average Score
Now, suppose you want more detail than just pass/fail. You score each answer on a scale from 1 to 5, where 5 is perfect and 1 is poor.
The scores for the 10 questions are:
s₁ = 4
s₂ = 5
s₃ = 3
s₄ = 4
s₅ = 5
s₆ = 2
s₇ = 3
s₈ = 4
s₉ = 5
s₁₀ = 3

Step 1: Add all the scores together.
Sum = s₁ + s₂ + s₃ + s₄ + s₅ + s₆ + s₇ + s₈ + s₉ + s₁₀
Sum = 4 + 5 + 3 + 4 + 5 + 2 + 3 + 4 + 5 + 3

Let us add them in groups to avoid errors:
4 + 5 = 9
9 + 3 = 12
12 + 4 = 16
16 + 5 = 21
21 + 2 = 23
23 + 3 = 26
26 + 4 = 30
30 + 5 = 35
35 + 3 = 38

The total sum of scores is 38.

Step 2: Divide the sum by N.
We know N = 10 (the number of questions).
Formula: Average Score = Sum divided by N
Calculation: 38 divided by 10 = 3.8

Result: The average score is 3.8 out of 5.

Summary of Results:
- Pass Rate: 60%
- Average Score: 3.8/5

These two numbers are your Minimum Evaluable Product for this evaluation set. They summarize the bot's performance on these specific 10 questions. If you improve the bot and run the same 10 questions again, you will get new values for P and the scores s₁...s₁₀. You can then recalculate to see if the pass rate or average score has increased.


## From scratch, in Python

This concept is algorithmic, so I will teach you the plain Python you need to implement a simple evaluation harness. You have never programmed, so let me introduce the first building block: a **list**. A list is a collection of items kept in order. You write it with square brackets, with commas between items. For example, `questions = ["What is the return period?", "Can I return a gift card?"]` is a list of two strings (text). A list can be changed after you make it. The other building block is a **loop**, specifically a `for` loop. A `for` loop goes through each item in a list, one at a time, and does something with it. For example, `for q in questions:` will run the indented lines once for each question in the list.

Here is the full script. It defines a list of test questions, a list of expected phrases, a list of bot answers (simulated here), and then it checks each answer for the expected phrase, counts passes, and calculates the pass rate.

```python
# Our evaluation set: questions and the phrase we expect in a good answer
test_cases = [
    ("What is the return period?", "30 days"),
    ("Can I return a gift card?", "not eligible"),
    ("Do I need the receipt?", "receipt"),
    ("What if the item is damaged?", "refund"),
    ("How long does a refund take?", "5 business days")
]

# Simulated bot answers — in reality these would come from your model
bot_answers = [
    "You can return items within 30 days of purchase.",
    "Gift cards are not eligible for return.",
    "Yes, you need the original receipt.",
    "Damaged items are eligible for a full refund.",
    "Refunds are processed within 5 business days."
]

# Step 1: count how many answers contain the expected phrase
pass_count = 0
total = len(test_cases)  # len() gives the number of items in the list

for i in range(total):
    question = test_cases[i][0]
    expected = test_cases[i][1]
    answer = bot_answers[i]

    if expected in answer:  # checks whether the phrase is present
        pass_count = pass_count + 1
        print(f"PASS: {question}")
    else:
        print(f"FAIL: {question} — missing '{expected}'")

# Step 2: compute the pass rate
pass_rate = pass_count / total
print(f"Pass count: {pass_count} out of {total}")
print(f"Pass rate: {pass_rate:.0%}")
```

Run this and you will see which questions pass and which fail, and the final pass rate. Change one of the `bot_answers` to something wrong, and you will see the pass rate drop. That is your measurement loop in action.


## The same thing, with a framework

In practice, an evaluation harness like this is often built on a framework such as MLflow, which provides a structured way to log metrics and compare runs. Here is a sketch of what the same evaluation might look like with MLflow (using `mlflow.evaluate`), though the underlying logic is still the Python you just wrote — MLflow is just wrapping it in a package that stores results for you.

```python
import mlflow

# Simulate an evaluation dataset and model predictions
# In reality, you would point MLflow at your model and a dataset
example_data = {
    "questions": ["What is the return period?", "Can I return a gift card?"],
    "predictions": ["You can return items within 30 days.", "Gift cards are not eligible."],
    "targets": ["30 days", "not eligible"]
}

with mlflow.start_run():
    mlflow.evaluate(
        data=example_data,
        model_type="text",
        targets="targets",
        predictions="predictions"
    )
```

Underneath, MLflow is doing the same kind of pass/fail or similarity checks you wrote by hand. The framework just adds bookkeeping: it records the metrics, the dataset, and the code version so you can compare runs later. The important habit is not the framework — it is that you built the harness at all.


## What the code does

Line by line:

- `test_cases` is a list of 5 tuples. A **tuple** is like a list but written with parentheses instead of square brackets, and it cannot be changed after creation. Each tuple holds two items: the question (a string) and the expected phrase. Strings are text values.
- `bot_answers` is a list of 5 strings, playing the role of your model's outputs.
- `pass_count = 0` initialises a counter.
- `total = len(test_cases)` — `len()` is a built-in Python function that returns the number of items in a list. Here it gives 5.
- `for i in range(total):` — `range(total)` produces the numbers 0, 1, 2, 3, 4. The loop runs once per number, and `i` takes each value in turn. This lets us visit each test case by its position.
- `test_cases[i][0]` — this reads the first item of the i-th tuple (the question). `[0]` is how you access an item in a collection, using its index. Indices start at 0, so `[0]` is the first, `[1]` the second.
- `if expected in answer:` — `in` checks whether one string appears inside another. If it does, we count a pass.
- `pass_count = pass_count + 1` — this adds 1 to the counter. It is the classic way to count in Python.
- The `print` statements use an f-string: a string preceded by `f` that lets you insert variable values into the text with curly braces. `{question}` gets replaced with the actual question text.
- After the loop, `pass_rate = pass_count / total` calculates the proportion. Note that dividing two integers in Python 3 gives a float (a number with a decimal point), so `6/10` becomes `0.6`, not `0`. That is what we want.
- `print(f"Pass rate: {pass_rate:.0%}")` formats the float as a percentage — `:.0%` turns `0.6` into `60%` with no decimals.

The logic is the heart of the MEP: a fixed set of inputs, a check, a count, and a rate.


## Common misconceptions

**The demo is good enough** — If I show my bot answering a few questions perfectly, that proves it works.

A demo shows curated examples that you chose because they work. It is a selection of successes, not a sample of reality. The bot might fail on 90% of real inputs, and the demo will never reveal that because you picked the 10% that pass. In fact: Build a fixed evaluation set with a spread of inputs — including edge cases and failures you expect — and measure the pass rate over that set. The number you get is far more honest than the demo.

**Smaller means fewer features** — The MEP is just a feature list cut to the bone.

An MEP is not about how many features you implement; it is about whether you can measure what you have. You could have one feature and still not be able to evaluate it if you have no test set and no scorer. In fact: The MEP is the smallest *deployable and measurable* slice. The measure is the non-negotiable part. If you cannot run it and get a score, it is not an MEP.

**Metrics are just decoration** — I can just eyeball the outputs and tell if it is good.

Eyeballing works for two examples and collapses at fifty. It is also not reproducible — another person will eyeball differently. Metrics give you a consistent, comparable number that can gate deployments and track regressions over time. In fact: Evaluation is a decision system, not a score. A metric that gates nothing is decoration. The MEP forces the metric to exist and to be used.

**The MEP is a one-time thing** — Once I have my evaluation set and my metric, I am done.

Your product changes — new features, new edge cases, new user behaviour — so your evaluation set must evolve too. If you never add cases, your metric stops being a predictor of real-world performance. Your library notes say datasets must evolve with the product, with changes versioned explicitly. In fact: Treat the evaluation set as a living artefact. Add new cases from real incidents, version the set, and recalibrate your judge periodically to avoid Evaluator Drift.

**More test cases is always better** — I should use thousands of test inputs to get a trustworthy score.

More inputs is good, but only if they are representative and labelled correctly. A thousand duplicated or irrelevant inputs give you a misleading score and cost time. The MEP is about the *minimum* number that gives you information you can act on. In fact: Start with a handful of diverse, hand-picked cases — enough to cover the main paths and edge cases. Grow the set as you learn where the system fails, but always keep it curated.


## In short

Build the smallest version of your system that can be measured, not merely demoed. That means a fixed evaluation dataset, a defined scoring method, and a metric you can compute before and after a change. The first milestone is informative, not impressive. Without measurement, you are at the mercy of the Evals Gap — regressions invisible until users complain. The MEP is the root of the entire LLM Evaluation tree.


## Where this is used

Where does the Minimum Evaluable Product show up in building modern AI systems? It is not a feature you ship; it is a discipline you follow. But every serious AI team uses it, sometimes without naming it, because they have been burned by the Evals Gap — shipping an LLM app with no measurement and discovering regressions only when users complain. Here is how the MEP manifests across the areas you asked about.

**Large language models (LLMs).** When OpenAI or Anthropic train a new model, they do not release it on the strength of a demo. They build a massive evaluation harness first — benchmarks like MMLU or HumanEval, plus internal gold sets — and they run every candidate model through it. That harness is the MEP scaled up. The smallest version is a handful of tasks and a score; the principle is identical. In practice, a developer at a company using an LLM will create a small evaluation set of, say, 50 prompts that represent their domain, and they will run every prompt through the model with a scoring rubric. The MEP forces them to decide the rubric *before* they change the model prompt or swap the model — that is the only way a before/after comparison means anything. Without it, they are guessing.

**Small language models on devices (TinyML, on-device).** When a model must run on a phone or a laptop, the constraints are different: size, latency, battery. The MEP still applies, but the measurement set must include those constraints. A developer building a keyboard autocomplete or a wake-word detector will build a small set of audio or text samples, run the model on the device (or a simulator), and measure not just accuracy but also latency in milliseconds and model size in megabytes. The smallest thing that can be measured here is a single voice command with a stopwatch — an MEP. The point is the same: you cannot know if a compression technique helped if you did not measure the original.

**Retrieval-augmented generation (RAG).** RAG systems fetch relevant documents and then generate answers from them. The failure modes are many: the retriever returns the wrong chunks, the generator ignores them, or the generated answer is not grounded in the retrieved text. A demo will not show those failures. An MEP for a RAG system is a set of questions, a known corpus, and two checks: (1) did the retriever return a document that contains the answer, and (2) did the generated answer, when checked against that document, stay faithful? You can measure the retrieval hit rate and the answer faithfulness score. The smallest version might be 10 questions and a simple keyword overlap check. Again, the MEP forces you to define 'good retrieval' before you tune anything.

**Agentic systems that plan and call tools.** Agents are the hardest to evaluate because they take many steps. An MEP here is the smallest multi-step task you can score. For example: 'Book a flight from Delhi to Mumbai on Tuesday, then confirm the price.' The agent must call a search tool, a booking tool, and reply with a confirmation. Your evaluation set might have three such tasks. Your scorer checks: did the agent call the right tools in the right order? Did the final message contain the confirmation? That is measurable. It is more complex than a chatbot single-turn check, but the principle holds: define the steps, the expected tool calls, and the final output format, and run it over a fixed set of tasks. Without that, agentic errors — looping, hallucinated tool calls, wrong order — are invisible until they cause real harm.

**Everyday applied machine learning (recommendation, vision, forecasting, fraud detection, speech).** The MEP is not specific to LLMs. In fraud detection, the smallest measurable product might be a single rule that flags transactions above a threshold, tested on a hundred labelled transactions, scored by precision and recall. In forecasting, the MEP is a single time series and a mean absolute error. In computer vision, a handful of images and an accuracy count. In speech, a set of audio clips and a word error rate. In every case, the discipline is identical: build the smallest test set and scorer *before* you invest in the fancy model. The first milestone is a metric, not a demo.

**What building software with this looks like in practice.** A developer reaches for a framework like MLflow (from your library's source, *Practical MLflow for Generative AI on Databricks*) or a library like `evaluate` from Hugging Face. They define an evaluation dataset as a CSV or a list of JSON objects — the test cases. They write scoring functions — deterministic validators for format checks, BLEU for reference overlap, an LLM judge for open-ended quality. They run the evaluation, see a score like 0.72, record it, then make a change — tweak a prompt, swap a model — and re-run to see if the score moves up or down. The framework records the metrics so they can compare runs across time. What breaks when this is done badly is the Evals Gap: teams ship prompts that work on their five favourite examples, celebrate a demo, and then production users see garbage because the five examples did not cover the real input distribution. The MEP exists to prevent exactly that.


---

Read this on the web: https://basavapurushottam.com/learn-ai/minimum-evaluable-product
