# Slice-Based Coverage

Evaluating on meaningful subsets — language, customer type, query category — rather than only on the aggregate.

**Rests on:** Four-Axis Quality, Safety, Cost and Reliability

*LLMs and Generative AI · LLM Evaluation*


## Why learn this

This concept is your first tool for making AI evaluation honest. You will build on it with Evaluation Datasets, Gold and Adversarial Sets, Regression Gates, and ultimately the Four-Axis Quality, Safety, Cost and Reliability framework. Without slice-based coverage, you will look at a single accuracy number, think everything is fine, and ship a system that fails for your most important users — the exact shape of a fairness incident. Mastering this unlocks the ability to ask 'where does this break?' instead of 'does this work overall?'


## The intuition

Imagine you are a teacher with a class of 100 students. You give them one test, and the average score is 85 out of 100. That sounds great. But hidden inside that average: 90 students scored between 90 and 95, and 10 students scored 10. The average is 85, yet those 10 students learned nothing. The average 'hid' a group that failed completely. Now imagine you are evaluating an AI system that answers customer questions in three languages — English, Hindi, and Tamil. You test it on 1,000 questions. It scores 90% overall. Great, right? But what if you split the results by language? Maybe it scores 97% in English, 95% in Hindi, and 40% in Tamil. That 40% is a crisis for your Tamil users, but the overall average never shows it. Slice-based coverage means you never look at just the single overall number. You always cut the results into meaningful groups — by language, by customer type, by query category — and check each group separately. The aggregate is a summary, but summaries lie. The slices tell the truth.


## An analogy

Think of a restaurant health inspector. They don't just walk in, taste one dish, and declare the whole restaurant safe. They check the kitchen, the storage, the hand-washing station, the refrigerator temperature — each area separately. One excellent dish doesn't excuse a filthy kitchen. The inspector works by slices, not by an aggregate 'restaurant score'. Now carry this to its limit: the inspector checks many areas because failure in one area has a different consequence than failure in another. If the fridge is broken, that's a different problem than the soup being too salty. Same with an AI system — a failure in the Hindi slice is a different problem than a failure in the English slice, and it needs a different fix. Where the analogy stops working: a restaurant inspector can physically inspect every area in an hour. An AI system might have millions of possible inputs, and you cannot slice every possible group. You must choose the slices that matter — the ones that mirror how your product is actually used. That choice is a judgment call, and it is exactly where the discipline lives. The inspector's job is comprehensive; your job is selective and deliberate.


## Definition

Slice-based coverage is the practice of evaluating an AI system on meaningful, predefined subgroups of its inputs — such as language, customer type, or query category — and reporting performance for each group separately, instead of relying only on an aggregate metric across everything.


## How it connects

You are starting fresh, so there is nothing to connect to yet. But know this: the library notes you will build reference close neighbours. Evaluation Datasets are the fixed set of inputs you run your slices on. Gold and Adversarial Sets extend this idea — gold sets check correct behaviour, adversarial sets probe failure; both are often analysed by slice. Regression Gates turn your per-slice metrics into a deployment decision: if the Hindi slice drops below 60%, do not ship. And the Four-Axis framework (Quality, Safety, Cost, Reliability) is where slice-based coverage gets applied — you slice each axis separately.


## The mathematics

This concept draws on arithmetic alone — division and percentages. The underlying branch of mathematics here is descriptive statistics: the practice of summarising a collection of numbers so you can reason about them. The specific operation you need is a group average: take a set of results, add them all up, and divide by how many there are. You have done this a thousand times — it is the mean. Slice-based coverage is not a new mathematical operation; it is a discipline about which means you compute. Instead of computing one mean over everything, you split the data into groups first, then compute a mean inside each group. The mathematics is just repeated division. What you need to internalise is the relationship between a group mean and the overall mean: the overall mean is the weighted blend of the group means — groups with more members pull the overall number harder. That is why a small group can be disastrous and still leave the overall number looking healthy.

Let S = a set of test inputs (a slice). Let C(S) = the number of those inputs the system answered correctly. Let N(S) = the total number of inputs in S. Then the slice accuracy is A(S) = C(S) / N(S) × 100%. The overall accuracy is A(all) = C(all) / N(all) × 100%. The relationship: A(all) = (sum over slices of A(S_i) × N(S_i)) / N(all).


## A worked example

```
You have 1,000 test questions. The system answered 900 correctly. You want to know how it performs per language. The breakdown is: 700 English questions (680 correct), 200 Hindi questions (180 correct), and 100 Tamil questions (40 correct).

Step 1: Overall: 900 correct / 1000 total = 0.90 = 90%   # The aggregate you would normally report. Looks great.
Step 2: English: 680 correct / 700 total = 0.971 = 97.1%   # Population slice: strong performance.
Step 3: Hindi: 180 correct / 200 total = 0.90 = 90%   # Second slice: matches the aggregate, healthy.
Step 4: Tamil: 40 correct / 100 total = 0.40 = 40%   # Third slice: catastrophic. The 90% overall hid this completely.
Step 5: Check: how does the overall 90% relate to the slices? (0.971 × 700 + 0.90 × 200 + 0.40 × 100) / 1000 = (679.7 + 180 + 40) / 1000 = 899.7 / 1000 ≈ 90%   # The overall is the weighted average of the slices. Bigger slices dominate — Tamil, with only 100 questions, barely moves the overall even though it is failing.

Result: The overall says 90% (acceptable). The Tamil slice says 40% (unacceptable). The aggregate hid the failure; slicing revealed it.
```


## From scratch, in Python

```python
# Slice-Based Coverage in plain Python

# Step 1: A list of test results. Each item is a tuple:
# (language, was_correct) where was_correct is True or False.
# A tuple is like a list but cannot be changed once created.

results = [
    ("english", True), ("english", True), ("english", False),
    ("hindi", True), ("hindi", True), ("hindi", False),
    ("tamil", False), ("tamil", False), ("tamil", True),
]

# Step 2: A dictionary to hold counts. A dictionary maps a key to a value.
# We want, for each language, the number of correct and total.
# We will build it with a loop.

# A dictionary literal with three languages, all starting at zero correct
# and zero total:
correct = {"english": 0, "hindi": 0, "tamil": 0}
total = {"english": 0, "hindi": 0, "tamil": 0}

# Step 3: Loop over every result. A for loop repeats the block for each item.
for lang, ok in results:
    total[lang] = total[lang] + 1
    if ok:
        correct[lang] = correct[lang] + 1

# Step 4: Compute and print per-slice accuracy.
# The .items() method on a dictionary gives us each (key, value) pair.
for lang, count in total.items():
    accuracy = correct[lang] / count * 100
    print(lang, "accuracy:", accuracy, "%")

# Output:
# english accuracy: 66.66... %
# hindi accuracy: 66.66... %
# tamil accuracy: 33.33... %
```


## The same thing, with a framework

```python
In practice, you would not write the loop by hand. You would use pandas, a library for working with tables of data. The same thing in a few lines:

import pandas as pd

# a DataFrame is a table with named columns
df = pd.DataFrame(results, columns=["language", "correct"])

# groupby groups rows by language, then we take the mean of the correct column
accuracy = df.groupby("language")["correct"].mean() * 100

print(accuracy)

# pandas does the same loop under the hood: it splits the rows into groups,
# then applies the mean (sum divided by count) inside each group.
```


## What the code does

Line by line: we start with a list of results — each entry is a pair (a tuple) of the language and whether the answer was right. We then create two dictionaries: one tracks how many were correct per language, one tracks how many were total per language. The first loop walks through every result, pulling out the language and the ok flag, increments the total count for that language, and if ok is True, increments the correct count too. The second loop walks through each language in the totals dictionary and computes accuracy by dividing correct by total and multiplying by 100. That division — correct divided by total — is the same arithmetic you did by hand in the mathematics section. The output shows each language separately, exactly what the aggregate would have hidden.


## Common misconceptions

**A high overall score means the system is good.** — If the overall accuracy is 95%, the system is working well. I only need to worry if the overall number is low.

The aggregate is a weighted average of the slices. A small slice with a 40% accuracy barely moves the average when a large slice is at 98%. The overall number can look excellent while a specific group — your Tamil users, your rural customers — is being failed completely. In fact: The overall score is only a summary of the per-slice scores. You cannot judge a system until you have seen the per-slice numbers. A system is only as good as its worst important slice.

**Slicing is just about fairness or ethics.** — Slice-based coverage is something I do if I care about being fair or avoiding a scandal, but it is not a technical necessity.

Slicing catches technical failures that have nothing to do with fairness: a rare product name that fails retrieval, a long plan that fails in agents, a low-budget phone that runs out of memory. These are engineering bugs, invisible in the aggregate, and slicing is the standard debugging tool. In fact: Slicing is a core engineering practice. It surfaces bugs and quality gaps across every dimension of your product, not just protected groups. Fairness is one beneficiary, not the whole story.

**If the slices are good, the model is good.** — Once I define slices and every slice scores high, I can be confident the system is production-ready.

You can only be confident about the slices you defined. An undiscovered slice — a new query category, a new language, a new customer segment — can fail completely and you would never know because you never looked. In fact: Slicing is a sample of the possible failures, not a proof of absence. Good slices reduce risk; they do not eliminate it. You pair slicing with monitoring in production and with adversarial sets that probe for unknown failures.

**More slices is always better.** — If slicing is good, then slicing by every possible dimension — language, device, time of day, gender, age, income — is even better. I should define dozens of slices.

Each slice needs enough data to be statistically meaningful. With 1,000 test questions and 40 slices, some slices will have only a handful of examples, and their accuracy numbers will be noise. Also, every slice you add is something you must maintain and make decisions on. In fact: Choose slices that mirror how the product is actually used and where failure matters most. A handful of well-chosen slices with enough data beats fifty thin slices you cannot trust.


## In short

Never judge an AI system by a single overall number. Split your test results into meaningful groups — by language, customer type, query category, or any dimension that mirrors real use — and measure each group separately. The aggregate hides failures in small groups; the slices reveal them. Choose slices deliberately, with enough data each to be meaningful, and let them gate your deployment decisions. A system is only as good as its worst important slice.


## Where this is used

Slice-based coverage is used everywhere in modern AI, and here is exactly how. In large language models like ChatGPT or Gemini, a vendor does not ship a model after one overall benchmark score. They slice by language — English, Hindi, Tamil, Swahili — because a model that excels on English but fails on Tamil is not production-ready for an Indian user base. They slice by query category: code generation, legal advice, medical questions, creative writing. Each slice gets its own accuracy number, and a model is only approved if every slice clears a threshold. That threshold decision is exactly the kind of judgment that separates a professional evaluation from a demo. In small language models running on a phone — think of an offline keyboard autocomplete or a voice assistant on a low-end Android device — the developer slices by device type and by input modality. A model that works on a flagship phone with 8GB of RAM may collapse on a 2GB budget phone, and the aggregate hides it. Slicing by device catches it. In retrieval-augmented generation, where a system first searches a knowledge base and then answers based on what it found, the failure modes are different: the retrieval step may fail for niche product names while succeeding for common ones. The developer slices by query category — common product, rare product, typo-laden query — and measures each separately. A retrieval failure on the rare-product slice is invisible in the overall, but it is the entire product for a user searching an obscure part. In agentic systems that plan and call tools — an AI that books flights, checks calendars, and sends emails — the developer slices by the tool being used and by the number of steps in the plan. A five-step plan may fail twice as often as a two-step plan, and that failure is hidden in the aggregate. Slicing by plan length reveals it, and the fix is different: it is not a model problem, it is an error-recovery problem. In everyday applied machine learning — a bank scoring loan applications, an e-commerce site recommending products, a fraud detector flagging transactions — slicing is the norm, not the exception. A credit model is sliced by income band and by region, because a model that works for urban customers may silently fail for rural ones — and that is precisely the shape of a fairness incident. A fraud detector is sliced by transaction amount: a model tuned for ₹10,000 transactions may flag or miss ₹100 transactions differently. What does a developer actually write? In code, they literally write the groupby call from the framework section: group by a column, compute the metric per group, and compare each group's number against a threshold. The library they reach for is pandas for the slicing, plus an evaluation harness like DeepEval or LangSmith that lets them define slice names and thresholds declaratively. What breaks when done badly: a mature AI product without slice-based coverage is an accident waiting to happen. The team reports a 94% accuracy at the quarterly review, the product launches, and the support inbox floods with complaints from one user segment the aggregate never showed. That is not a hypothetical — it is the documented shape of multiple real-world fairness scandals. The fix is not more data or a better model; it is a better evaluation practice. Every major evaluation framework — from the academic HELM benchmark to OpenAI's evals library — treats per-slice reporting as table stakes. If this concept is missing from your evaluation, you are not evaluating; you are averaging.


---

Read this on the web: https://basavapurushottam.com/learn-ai/slice-based-coverage
