# Offline vs Online Evaluation

Offline evaluation runs against fixed datasets before release; online evaluation measures real traffic after it.

**Rests on:** Evaluation Datasets

**Leads to:** Training-Time vs Inference-Time Evaluation

*LLMs and Generative AI · LLM Evaluation*


## Why learn this

You need to know the difference between testing a model in a sandbox and watching it work in the real world because this distinction protects your product from silent failures. It helps you answer two critical questions: 'Did this change make things better?' and 'Is it safe to show to users?' By understanding these phases, you gain control over when new features go live and how you spot problems after they launch. Without this knowledge, you might release a change that looks good in tests but breaks the user experience, or you might miss a slow decline in quality because your testing method was blind to real-world nuances. You are building a system that ensures every update adds value rather than risk.


## The intuition

Imagine you are launching a new feature in an app. Offline evaluation is like a controlled lab test. You take the new code and run it against a fixed set of 1,000 example questions with known correct answers. This happens before any user sees it. It is fast because you only run those specific 1,000 cases; it is safe because no real person is affected if it fails; and it is repeatable because you can run the exact same test again tomorrow to see if things got worse. However, these 1,000 examples might not cover every weird edge case a real user throws at your system. So, the test might pass with flying colors, yet fail in practice. This leads to Evaluator Drift: imagine if the person writing the correct answers for your test set changes their mind next month about what counts as 'good.' Suddenly, your test results no longer match reality, and you can't compare today's score to last month's because the ruler has changed. Online evaluation is like opening the feature to 1% of your users. Now you see how it performs with real language, real ambiguity, and real frustration. It gives you the true truth about user satisfaction, but every mistake is visible to a live person, which can be costly or damaging if a bug slips through. The best strategy uses both: you use the offline test as a hard gate—no deployment if the score drops below 95%—but you also monitor the online reaction closely because only the real world shows you the full picture.


## An analogy

Think of a professional chef creating a new dish for a restaurant menu. Offline evaluation is the chef cooking the dish in the private kitchen, tasting it themselves, and checking it against a strict recipe card that specifies exact amounts of salt, cooking time, and plating style. This is safe and cheap; if the dish tastes bad, you throw it away without losing any customers. But the chef knows their own palate might differ from the guests'. If the head chef changes the recipe card next week to be stricter on spice levels, the old 'passed' scores are now meaningless—that is Evaluator Drift: the standard for 'good' has shifted, so old results don't predict new success. Online evaluation is serving that dish to a small group of actual diners and watching their plates return clean or empty, reading their reviews, and noting if they ask for more sauce. This provides the real truth about whether people will buy it. However, one bad batch can ruin a dinner service. The kitchen uses the private test to filter out clearly bad recipes (the gate), then serves a few portions online to gauge reaction. The analogy breaks down in speed: a restaurant cannot instantly revert a dish once served, but software can roll back changes in seconds, making the combination of strict offline gates and rapid online monitoring much more powerful than kitchen management.


## Definition

Offline evaluation is measuring a model's accuracy using a static set of past examples before release, while online evaluation is measuring its performance by observing how real users interact with it after deployment.


## How it connects

This concept pairs directly with Evaluation Datasets: you need a fixed, unchanging collection of questions to make offline testing reliable and comparable over time. It connects to Model Monitoring: once the model is live, online evaluation provides the continuous stream of data that monitoring tools analyze to detect performance drops or Evaluator Drift.


## The mathematics

The mathematics here is not about fancy formulas — it is about counting and comparing. The only operations you need are ratios, percentages, and simple averages, all of which you already know from everyday life. For instance, if you want to know how often a model's answer was judged correct in an offline test, you count the correct ones and divide by the total number of test questions. That fraction, expressed as a percentage, is your offline accuracy. The same idea works for online evaluation: you count how many live requests resulted in a 'thumbs up' or a successful task, and divide by total requests. The mathematics is really about precision and repeatability: offline scores are stable because the dataset never changes, so you can compare two models exactly; online scores fluctuate because the traffic is different every hour, so you need to average over a large sample before you trust the number. There is no calculus here, no algebra beyond dividing one number by another — the challenge is not in the arithmetic but in deciding what to count and how to weight the results.

Let T be the total number of test items in an evaluation. Let C be the number of items judged correct or successful. Then the accuracy (or success rate) is:

accuracy = C ÷ T

For online evaluation, T and C are counts from live traffic over a fixed time period. The symbol 'C' stands for 'correct count', 'T' for 'total count'. There is no other formula needed here — the real mathematics is in choosing the metric and the threshold, not in the arithmetic.


## A worked example

```
You are evaluating a customer-support chatbot. You have two versions: the current one (A) and a candidate improved one (B). You first run both on a fixed offline dataset of 200 real customer questions, where each answer is manually labelled as 'good' or 'bad'. Then you do an online test where you show B to 10% of live users for a week and measure how many of their questions end with a 'satisfied' rating.

Step 1: Offline A: 160 good out of 200 → accuracy = 160 ÷ 200 = 0.80 = 80%   # The fraction of correct answers in the fixed set.
Step 2: Offline B: 170 good out of 200 → accuracy = 170 ÷ 200 = 0.85 = 85%   # B seems better, but only on this fixed set.
Step 3: Online A (baseline week): 1200 satisfied out of 1500 total → 1200 ÷ 1500 = 0.80 = 80%   # The current version's real-world satisfaction rate.
Step 4: Online B (test week): 1450 satisfied out of 1800 total → 1450 ÷ 1800 ≈ 0.806 = 80.6%   # The new version's live satisfaction rate — only a tiny 0.6% improvement, within noise.
Step 5: Compare: offline said +5% (85 vs 80), online said +0.6% (80.6 vs 80)   # The offline gain did not survive real traffic — the fixed set was too easy or unrepresentative.

Result: Offline evaluation suggested B was a clear win (85% vs 80%), but online evaluation showed the real improvement was negligible. Without the online check, you would have shipped B believing it was a major improvement. The offline test is necessary to weed out failures cheaply, but it is not sufficient to declare success.
```


## From scratch, in Python

```python
# Python building-blocks used here:
# - list: an ordered collection of items, e.g. ['cat', 'dog', 'goldfish']
# - for loop: repeats a block of code over each item in a list
# - if/else: chooses between two actions based on a condition
# - function: a named block of reusable code (introduced below)
# - dictionary: a lookup by name, e.g. {'a': 80, 'b': 85} (introduced below)

# ---------- Part 1: Offline evaluation on a fixed dataset ----------

# We have a fixed list of reference answers (the 'gold set').
# Each is a dictionary with the question and the expected answer.
# A dictionary stores pairs of a key (the name) and a value (the data).
# We look up the value by its key: reference['expected'] gives the expected answer.
reference_set = [
    {'question': 'What is the refund policy?', 'expected': 'Full refund within 30 days'},
    {'question': 'How do I reset my password?', 'expected': 'Go to settings, click reset, follow the email'},
    {'question': 'Do you ship to India?', 'expected': 'Yes, we ship to India with 5-7 day delivery'},
    # ... imagine 197 more here ...
]

# We also have a candidate model's answers (imagine these come from running the model).
candidate_answers = [
    'Full refund within 30 days',
    'Go to settings, click reset, follow the email',
    'We ship to India in 5-7 days',
    # ... and so on, one per question ...
]

# This function counts how many candidate answers exactly match the expected answer.
# - len(list) gives the number of items in a list.
# - range(len(reference_set)) gives the numbers 0, 1, 2, ... up to the last index.
# - reference_set[i]['expected'] gets the i-th reference's expected answer.
# - candidate_answers[i] gets the i-th candidate answer.
# - The 'for i in ...' loop visits each index one at a time.
def offline_accuracy(reference_set, candidate_answers):
    correct = 0
    total = len(reference_set)
    for i in range(total):
        if candidate_answers[i] == reference_set[i]['expected']:
            correct = correct + 1
    return correct / total

# Run it on a tiny subset (just the 3 shown above for demonstration;
# in reality you would run on all 200).
tiny_ref = reference_set[0:3]   # first 3 items (list slicing: from index 0 up to, not including, 3)
tiny_cand = candidate_answers[0:3]
print('Tiny offline accuracy:', offline_accuracy(tiny_ref, tiny_cand))
# Output: Tiny offline accuracy: 1.0 (all 3 matched)

# ---------- Part 2: Online evaluation from live traffic logs ----------

# Live traffic is a list of dictionaries, each with the user's question,
# the model's answer, and whether the user clicked 'helpful' (1) or 'not helpful' (0).
# Each entry looks like: {'question': '...', 'answer': '...', 'helpful': 1}
# (In practice, this list is streamed from a server, but here we create a sample.)
live_logs = [
    {'question': 'What is the refund policy?', 'answer': 'Full refund within 30 days', 'helpful': 1},
    {'question': 'How do I reset my password?', 'answer': 'Go to settings, click reset, follow the email', 'helpful': 1},
    {'question': 'Do you ship to India?', 'answer': 'We ship to India in 5-7 days', 'helpful': 1},
    {'question': 'What is your return policy?', 'answer': 'We accept returns within 14 days', 'helpful': 0},
    # ... imagine thousands more ...
]

# This function computes the success rate as the fraction of helpful marks.
def online_success_rate(logs):
    helpful = 0
    total = len(logs)
    for entry in logs:
        if entry['helpful'] == 1:
            helpful = helpful + 1
    return helpful / total

print('Online success rate:', online_success_rate(live_logs))
# Output: Online success rate: 0.75 (3 out of 4 helpful)

# ---------- Part 3: A simple regression gate ----------

# A regression gate decides whether a candidate is good enough to ship.
# It compares the candidate's offline accuracy to the current model's accuracy,
# and only ships if the candidate is at least as good (and ideally better).
# Here is a function that encodes that decision:

def should_ship(candidate_accuracy, current_accuracy, target_accuracy):
    if candidate_accuracy >= target_accuracy and candidate_accuracy >= current_accuracy:
        return True
    else:
        return False

# Example: target is 80%, current is 80%, candidate is 85%
print(should_ship(0.85, 0.80, 0.80))  # True

# Example: target is 85%, current is 80%, candidate is 82%
print(should_ship(0.82, 0.80, 0.85))  # False
```


## What the code does

The code is split into three parts. Part 1 defines a `reference_set`, which is a list of dictionaries — each dictionary holds a question and the expected answer. This is your fixed offline dataset. The `candidate_answers` list holds the model's answers in the same order. The `offline_accuracy` function takes both lists, loops over each index, and counts how many candidate answers exactly match the expected answer. It returns the fraction `correct/total`. In the demo, we only use the first 3 items to see the output; in reality, you would feed the full 200. Part 2 creates `live_logs`, another list of dictionaries, but now each entry also has a `helpful` field with 0 or 1. The `online_success_rate` function loops through each log entry, counts the `helpful == 1` entries, and returns the fraction. Part 3 defines `should_ship`, a simple rule: only ship if the candidate's accuracy is at least the target AND at least the current model's accuracy. This is the essence of a regression gate — it turns a metric into a go/no-go decision.


## Common misconceptions

**Offline accuracy is the true measure of quality** — If my model gets 98% on my test set, it will be great for real users.

Offline sets are fixed and finite. Real users ask questions you never imagined, and the distribution of live traffic is different from your curated set. The loan-officer analogy breaks here: the rehearsal can't cover every audience reaction. In fact: Offline accuracy is a proxy, useful for filtering, but the only true measure is how it performs on live, representative traffic.

**Online evaluation is too risky, so we just do offline** — We can't risk bad experiences for users, so we'll only test offline and ship if it looks good.

If you never run online tests, you never know what actually works. You can mitigate risk by limiting online tests to a small traffic percentage or to a specific segment, and by having rollback ready. In fact: Online evaluation is the only way to measure real-world performance; it should be run on a small, controlled slice of traffic before full rollout.

**A/B testing is the same as online evaluation** — As long as we compare two versions on live traffic, we have done online evaluation.

A/B testing is one method of online evaluation, but it often suffers from small sample sizes, short durations, and selection bias. Online evaluation also includes monitoring dashboards, user feedback analysis, and drift detection. In fact: A/B testing is a common tool within online evaluation, but a full online evaluation also tracks ongoing metrics and detects regressions over time.

**Offline eval is only about accuracy** — The only metric that matters in offline evaluation is how many answers are correct.

Accuracy is one metric, but it ignores cost, latency, safety, and reliability. A model can be 95% accurate but produce a harmful answer that a user sees; that is a failure of safety, not captured by accuracy. In fact: Offline evaluation should measure multiple axes: quality (accuracy), safety, cost, and reliability, each with its own threshold.

**Once you ship the model, offline eval is done** — We ran offline tests, it passed, so we are done with evaluation until the next release.

Real-world traffic changes; the model can drift as user behaviour shifts. You need continuous online monitoring to catch when performance degrades. In fact: Online evaluation is a continuous process, not a one-time event. It feeds feedback loops back into retraining.


## In short

Offline evaluation runs against fixed datasets before release — fast, safe, repeatable, but not fully representative of reality. Online evaluation measures real traffic after deployment — authoritative, but slower and carries user-facing risk. Mature systems use offline to gate what reaches online, and online to confirm the true effect. The arithmetic is simple (correct ÷ total), but the real work is choosing the right metrics and thresholds, and always verifying offline results with an online test.


## Where this is used

This concept underpins nearly every serious AI deployment. In large language models (LLMs), offline evaluation uses curated benchmark sets — like MMLU, HumanEval, or a company's own gold set — to compare model versions before release. A developer runs the candidate model on the fixed set, computes accuracy, and gates whether it can enter a shadow deployment. Online evaluation then measures real user interactions — such as thumbs-up/down, rate of regen requests, or task completion — to confirm the offline gain holds in the wild. For small models running on-device (e.g., a phone assistant), offline evaluation is often run on a compressed dataset of edge cases, but online is critical because device-specific quirks (different hardware, user accents, network variability) are impossible to replicate offline. In retrieval-augmented generation (RAG), offline evaluation tests whether the retriever finds the right documents and whether the generator answers correctly given those documents — often with recall@k for retrieval and answer correctness for the generator. Online evaluation tracks whether users actually click on cited sources or whether the assistant's answers lead to successful task completion. In agentic systems that plan and call tools, offline evaluation simulates the full agent loop against a fixture (a scripted environment of fake tool responses) to check that the agent takes the correct sequence of actions. But because the real world's tool outputs are unpredictable, online evaluation runs the agent on a limited percentage of live traffic while monitoring for failures like tool call errors, loops, or unsafe actions. In everyday applied ML — recommendation, forecasting, fraud detection — offline evaluation is the standard train/test split, and online is A/B testing; a fraud model, for instance, is trained offline on historical transactions, but its real performance is measured by the rate of false positives and false negatives in live approval decisions. A developer working with any of these systems typically writes code like the `offline_accuracy` and `online_success_rate` functions above, and then wraps the decision in a regression gate. When done badly, teams skip online evaluation entirely and deploy based on offline metrics alone, leading to models that perform well on benchmarks but fail with real users. When done well, the offline gate catches obvious regressions cheaply, and the online test provides the final, trusted verdict.


---

Read this on the web: https://basavapurushottam.com/learn-ai/offline-vs-online-evaluation
