# Benchmarking

Comparing models on standardised task suites to get a common frame of reference.

**Rests on:** Evaluation Datasets

**Leads to:** Canary Releases, LLM as Judge, LLM Safety Benchmarks, Trust Rating Scales

*LLMs and Generative AI · LLM Evaluation*


## Why learn this

Benchmarking is the measuring stick of AI. When someone says ‘this model is better than that one’, they are almost always pointing at a benchmark score. If you want to know which model to choose for your task, whether a new version is actually an improvement, or how to talk about AI performance with any credibility, you need to know what those numbers mean — and what they don't. This page gives you that foundation, so that when you later work with evaluation, safety benchmarks, or model monitoring, you already know the ground rules of comparison.


## The intuition

Suppose you want to buy a new phone. You have two models in mind, and you want to know which one takes better photos. You could just try them both yourself, but that takes time and might be biased by your own preferences. Instead, you look at a photo review site that takes a standard set of pictures — a sunlit landscape, a dim indoor shot, a portrait — with each phone, and then compares the results. That standard set of pictures is a benchmark. It gives both phones the same task, under the same conditions, so you can compare them fairly. In AI, a benchmark is exactly that: a fixed, standardised set of tasks that many different models are asked to perform, so that their results can be compared. The score a model gets on that benchmark is its report card — but report cards only measure what the test actually covers. If the test is all about maths, you don't learn which student writes better essays. In the same way, a benchmark that tests one kind of task tells you nothing about performance on a different kind of task.


## An analogy

Think of benchmarks as international sports competitions, like the Olympics. Every athlete from every country competes in the same events, under the same rules, and their performances are measured with the same stopwatches and tape measures. This gives a common frame of reference: a gold medal in the 100-metre sprint means the same thing whether the winner is from India or Brazil. That is exactly what benchmarking does for AI models. The model is the athlete, the test tasks are the events, and the score is the medal. But the analogy has an important limit: athletes train specifically for their events. A sprinter who wins gold in the 100-metre does not automatically win the marathon. In AI, a model that scores brilliantly on a benchmark may perform poorly on a real-world task that is slightly different in style or context. Moreover, athletes and coaches study past Olympics to train for the next one — and if the same events are reused year after year, competitors start to train specifically to game the test, rather than to improve their true ability. That is contamination: when a model has seen the benchmark questions during its training, its score stops being an honest measure and becomes a memory test. So the Olympic analogy holds until the point where the test itself becomes part of the training — and then it breaks down, because the benchmark no longer measures what it claims to measure.


## Definition

A benchmark is a fixed, standardised suite of tasks, each with a known or expected answer, used to compare the performance of different AI models, but its score only reflects performance on that specific set, not on your real-world problem.


## How it connects

This page is your first step into the wider topic of LLM Evaluation — the art of measuring whether a generative or agentic AI system actually works. The learner's library records that benchmarking sits under that parent concept, alongside neighbouring topics like BLEU and Reference Metrics (which are automatic scoring methods used inside benchmarks), Evaluation Datasets (the curated collections of examples that benchmarks are built from), and LLM Safety Benchmarks (specialised suites for harmful behaviour). Once you understand benchmarking, you'll be ready to dive into those. A benchmark is essentially a pre-made evaluation dataset with a scoring rule attached. Everything you learn here — contamination, the gap between benchmark and real task, and the importance of your own evaluation — will apply directly when you build your own sets.


## The mathematics

The mathematics of benchmarking is not about advanced calculus or algebra — it's about the simple arithmetic of counting and averaging. When a model is tested on a benchmark, it is given a series of tasks. For each task, the model produces an answer. That answer is compared against the correct answer (or against a reference answer, if the task is open-ended like writing a sentence). The comparison either yields a score for that task (like '1' for correct, '0' for incorrect, or a partial score like 0.7 for a partially correct answer). Then, to get an overall benchmark score, you simply average all those individual task scores — you add them all up and divide by the number of tasks. That average is the single number you see quoted in headlines ('Model X scores 88 on SuperGLUE'). So the only arithmetic you need is: count how many tasks the model got right (or how much credit it earned), and divide by the total number of tasks. That's it. There is no deeper maths hiding underneath — the sophistication is all in the choice of tasks and the scoring rule, not in the final calculation.

To write this down without using symbols that might confuse you, we describe the process in words:

1. Take the score from the first task.
2. Add the score from the second task to it.
3. Keep adding the score from each new task until you have added every single task's score together. This total is called the sum of the scores.
4. Count how many tasks there were in total. We will call this number N (read as 'N').
5. Divide the sum of the scores by N.

The result of this division is the overall benchmark score.


## A worked example

A benchmark has 5 tasks. A model is tested on each. For each task, it gets a score between 0 and 1 — 1 means fully correct, 0 means fully wrong, and values in between mean partial credit. The model's scores on the 5 tasks are: 0.8, 1.0, 0.6, 0.9, and 0.7.

Step 1: Add up all the individual task scores to get the total.
   Start with the first score: 0.8
   Add the second score: 0.8 + 1.0 = 1.8
   Add the third score: 1.8 + 0.6 = 2.4
   Add the fourth score: 2.4 + 0.9 = 3.3
   Add the fifth score: 3.3 + 0.7 = 4.0
   The sum of the scores is 4.0.

Step 2: Divide the total by the number of tasks to get the average.
   There are 5 tasks, so we divide 4.0 by 5.
   4.0 ÷ 5 = 0.8

Result: The model's overall benchmark score is 0.8, or 80%. That's the single number that gets quoted as 'the model's score'.


## From scratch, in Python

```python
from statistics import mean

# Our benchmark: a list of tasks. Each task is a tuple (input, expected_answer).
# A tuple is like a list, but it can't be changed after it's made.
# The first element is the input, the second is the expected answer.

benchmark = [
    ("2 + 2", "4"),
    ("What is the capital of France?", "Paris"),
    ("Translate 'hello' to Hindi", "नमस्ते"),
    ("What is the largest planet?", "Jupiter"),
    ("What is the speed of light?", "299,792,458 m/s")
]

# A model is a function that takes an input string and returns an answer string.
# We'll define a very simple, not-very-good model: it always says "I don't know".

def dumb_model(input_text):
    return "I don't know"

# A slightly better model: it knows a few facts.

def better_model(input_text):
    if input_text == "2 + 2":
        return "4"
    elif input_text == "What is the capital of France?":
        return "Paris"
    else:
        return "I don't know"

# Now we write a function that evaluates a model on the benchmark.
# It will return the average score.

def evaluate_model(model, tasks):
    scores = []  # an empty list to hold the score for each task
    for input_text, expected in tasks:
        # Ask the model for its answer
        response = model(input_text)
        # Compare with the expected answer. We'll use exact string matching.
        # If they match, score 1; if not, score 0.
        if response == expected:
            scores.append(1)
        else:
            scores.append(0)
    # Now compute the average of all the scores
    return mean(scores)

print("Dumb model score:", evaluate_model(dumb_model, benchmark))
print("Better model score:", evaluate_model(better_model, benchmark))
```


## What the code does

This code builds a tiny benchmark from scratch, showing you exactly what a benchmark is and how evaluation works. Here's what each part does:

**The benchmark list** – We create a list called `benchmark`. A list holds items in order, and you can change it later. Each item in the list is a *tuple* — a pair of values: the first is the question, the second is the expected answer. A tuple is like a list but can't be changed after creation. So the benchmark has five tasks.

**The models** – We define two simple models as *functions*. A function is a block of reusable code that takes an input and returns an output. The first model, `dumb_model`, always returns the string `"I don't know"` regardless of input. The second, `better_model`, has a couple of `if`/`elif` statements — these check the input against specific strings and return a known answer if it matches, otherwise it also says "I don't know". This is clearly a toy, but it shows the structure.

**The evaluation function** – The `evaluate_model` function takes a model and a list of tasks. Inside it, we start with an empty list `scores`. Then we loop over each task using a `for` loop. For each task, we call the model on the input, get its response, and compare it to the expected answer using `==`. If they match, we append `1` to the scores list; otherwise `0`. After the loop finishes, we use `mean(scores)` from the `statistics` module to average the scores.

**Printing results** – Finally, we call `evaluate_model` with each model and print the result. The dumb model gets 0.0, and the better model gets 0.4 (two out of five correct). This is a microcosm of real benchmarking: define tasks, run a model, score each, average.


## Common misconceptions

**Benchmark Score = Real-World Performance** — If a model scores 90% on a popular benchmark, it will perform 90% as well on my task.

A benchmark is a specific set of tasks with specific expected answers. Your real-world task might be completely different in style, difficulty, or domain, so the model's performance there could be much worse (or better). Also, many public benchmarks are contaminated — the model's training data may have included the test questions, inflating the score. In fact: A benchmark score is an indication of performance on that specific benchmark, not a guarantee of performance elsewhere. Always evaluate the model on your own, customized test set that reflects your real task.

**Higher Benchmark Score is Always Better** — The model with the highest benchmark score is always the best model to use.

The best model depends on what you care about: accuracy, speed, cost, safety, and even memory usage. A model that scores 1% lower on accuracy but runs twice as fast on your phone may be the better choice. A benchmark hides these trade-offs. In fact: Consider all relevant factors — including speed, cost, and safety — in addition to the benchmark score. The right model for your situation balances all of them.

**Benchmarks Are Permanent** — Once a benchmark is established, it stays valid forever.

As models improve and are trained on more data, they may end up 'cheating' on old benchmarks — the test questions get memorized during training, so the score stops reflecting actual reasoning. This is known as contamination. In fact: Benchmarks must be refreshed regularly, and you should check whether your model's training data includes benchmark questions. Public benchmarks have a shelf life.

**Benchmark Comparisons Are Always Fair** — If two models are both evaluated on the same benchmark, their scores can be directly compared without any caveats.

Even with the same benchmark, the evaluation setup can vary: different prompting styles, different decoding parameters (like temperature), or different amounts of test-time compute can change the scores. Also, some models may have been fine-tuned on the benchmark itself. In fact: When comparing scores, ensure the evaluation protocols are identical (same prompts, same decoding parameters) and check whether the benchmark was included in the model's training data.

**Benchmarking Is Just About Accuracy** — A benchmark always measures how many questions the model gets right.

While many benchmarks use accuracy, others use other scoring rules: e.g., F1 score (harmonic mean of precision and recall), BLEU score (n-gram overlap for generated text), or task-specific metrics like exact-match or average precision. These metrics can give very different pictures of performance. In fact: Understand what metric the benchmark uses — it might measure faithfulness, fluency, or how well the model's answer uses the given context, not just whether it's factually correct.

**A Good Benchmark Guarantees a Good Model** — If I have a good benchmark, then any model that scores well on it will work well in production.

A benchmark is only a proxy for the real world. The benchmark tasks may be too simple, too narrow, or too different from real-world usage. Also, your production environment may have latency constraints, safety requirements, or user feedback loops that the benchmark doesn't capture. In fact: Use benchmarks as a first screening, but always run a pilot or A/B test with real users to validate that the model actually solves your problem.


## In short

Benchmarking means comparing models on a fixed, standardised set of tasks to get a common frame of reference. The score is just an average of task scores, but it only measures the benchmark, not your real task. Always build and run your own evaluation.


## Where this is used

Benchmarks are everywhere in AI, but they're deceptively simple. Here are the key places you'll meet them.

**In large language models (LLMs)** – When a company releases a new LLM, they publish its scores on public benchmarks like MMLU (massive multitask language understanding), GSM8K (math problems), and SuperGLUE (a suite of language understanding tasks). These scores are how the model is advertised — GPT-4 scores X, Gemini scores Y — and how researchers compare progress. But each benchmark is just a fixed list of questions with known answers, and the model is scored by how often it gets the answer right. Underneath, the model is just producing text, and the evaluator checks it against the reference answer. Developers also build their own internal benchmarks from their own test sets, to track improvements as they train new versions.

**In small models running on phones or laptops** – Small language models (like those used for on-device AI) are also benchmarked, but the focus shifts to computational efficiency alongside accuracy. Benchmarks might measure how fast the model responds, how much memory it uses, and whether it can run on a low-powered chip. The model's accuracy might be measured on a subset of a public benchmark, but the key question is whether it delivers acceptable quality within tight resource limits. A developer might use a speed benchmark alongside an accuracy benchmark to decide if the model is viable for a phone.

**In retrieval-augmented generation (RAG)** – RAG systems retrieve relevant documents to help the model answer questions. Measuring the quality of a RAG system is trickier: you need to know whether the right documents were retrieved (measured with retrieval metrics like recall@k, which checks whether the correct document is in the top-k results) and whether the final answer is correct. The benchmark would consist of questions, a set of documents to search through, and the expected answers. Each step — retrieval and generation — gets its own benchmark score. For example, you might have a set of 50 queries, each with a known relevant document, and you measure whether that document appears in the top 5 retrievals.

**In agentic systems** – Agents that use tools and take multi-step actions are harder to benchmark because there's no single 'correct answer'. Instead, the benchmark might be a set of tasks like 'book a flight from Delhi to Mumbai on June 3rd', with a checklist of required steps (departure city, arrival city, date). The agent's performance is scored by how many of the required steps it completed successfully. These benchmarks often involve a simulated environment with tools to call and a small API to interact with. The score is then a pass/fail on each sub-step, averaged across tasks.

**In everyday applied machine learning** – In any ML project — fraud detection, recommendation, forecasting — a benchmark is just your test set with an evaluation metric. You split your data into training and test, train the model on the training set, and then evaluate it on the test set. The test set is the benchmark, and your accuracy or F1-score is the score. When you try a different model, you evaluate it on the same test set to compare. This is the bedrock of machine learning practice, and understanding benchmarks prevents you from comparing models trained and evaluated on different data.

**What developers actually write** – A developer working with an LLM will often use a framework like Hugging Face's `evaluate` library or the `lm-evaluation-harness` to run a model through a public benchmark. But more importantly, they'll write their own evaluation script that loads a curated set of test examples, runs the model, and computes scores. They might use a library like `pytest` for deterministic checks, or a simple script like the one in the code section. The key is that the test set is fixed and versioned — if it changes, the scores aren't comparable. When done badly, the benchmark is contaminated (the model has seen the test answers), or the test set doesn't match the real task, or the scores are prone to random variation because the model is stochastic (producing different answers each run). A developer might also track benchmark scores over time to catch regressions: if the score drops, something went wrong.


---

Read this on the web: https://basavapurushottam.com/learn-ai/benchmarking
