# Skill Testing and Evals

Applying evaluation discipline to reusable agent skills, checking they behave correctly across intended cases.

**Rests on:** Evaluation Datasets, LLM as Judge

*LLMs and Generative AI · LLM Evaluation*


## Why learn this

You are building automated assistants that perform tasks on your behalf. If one of these helpers makes a quiet mistake, the entire workflow fails without warning. This concept teaches you how to verify that a helper actually does what it claims before you let it go, and how to catch the moment a software update silently ruins a task that used to work perfectly. It is the foundation for checking if an answer fits the request, whether checks run consistently, and how to build systems that fail loudly so you can fix them quickly. For example, knowing that a skill has passed 95 out of 100 test cases gives you concrete confidence; failing 8 out of 10 tells you exactly where it breaks.


## The intuition

Imagine you have written down a recipe for the perfect cup of chai. You hand it to a friend. They follow every step exactly as written, but the result is 'fine but not great.' You taste it and realize it is too sweet. Your recipe works when *you* make it because you intuitively know how big a 'pinch' of salt is or what 'simmering for a while


## An analogy

Think of a quality inspector on a factory line. The inspector does not make the product; they check a sample of what comes off the line. They have a checklist: 'Does the lid fit? Does the label face front? Does the button click?' If the sample passes the checklist, the line is trusted to keep running. If it fails, the line stops and someone investigates. That is exactly what skill testing is. Your skill is the factory line; the model that runs the skill is the worker; the checklist is your eval set. You run the skill on a set of pre-chosen inputs, check the outputs against pre-chosen expected answers, and decide whether the skill is good enough to be used. The analogy holds further: a good inspector does not just check normal products — they deliberately test edge cases where things might go wrong, such as a lid that fits tightly because the jar is slightly deformed or a label that smudges if the ink cartridge is low. In skills testing, these are inputs designed to break the logic, like an empty email request or a thousand-word ramble with no clear instruction. Where the analogy stops: a factory inspector can check every product. An AI skill is infinite — you cannot test every possible input, so you sample. And unlike a factory, your worker (the model) can change without warning. A new model version might behave differently on the same input. So you re-run the checklist every time the model changes, not just when the skill changes. That is the part of the analogy that is uniquely true for AI: the factory line itself is alive and shifting beneath you.


## Definition

Skill testing and evals is the practice of running a reusable agent capability against a fixed set of carefully chosen inputs, comparing the results to expected behavior, and using the pass or fail rate to decide whether the capability is safe to use and when it has broken.


## How it connects

This is your first step into LLM evaluation, the parent concept in your library that covers how to judge if a model is doing what you want. You have not yet met its neighboring topics — answer relevancy (how well the output matches the user's specific question), deterministic validators (rules that check for obvious errors like formatting or missing data without ambiguity), and gold sets (a curated list of perfect answers used as the standard for comparison). The library tells me you've noted: 'Evals are the objective test for a skill just as for a model.' That is the core idea we will make real here. You also noted 'Scope should be tested at the edges, where skills degrade first' — that's the second half of this page. And: 'Untested skills accumulate silent breakage as models change beneath them' — that's the why.


## The mathematics

This concept uses only basic arithmetic: counting whole numbers and dividing them to find a percentage. There is no algebra, no variables that stand for unknowns, and no complex formulas. The mathematics here is simply a ratio.

The core operation is the pass rate. The pass rate tells you what fraction of your tests were successful. To find it, you take two counts:
1. The number of inputs that produced the correct result (the passing count).
2. The total number of inputs you tested (the total count).

You divide the passing count by the total count. This gives you a decimal between 0 and 1. To make it easier to read, you multiply that decimal by 100 to get a percentage.

Pass rate = (passing count divided by total count) multiplied by 100.

You will also use subtraction and comparison. If you change your model or your instructions, you run the same test again. You compare the new pass rate to the old one. If the new number is lower, the change made things worse. If it is higher, the change helped. The mathematics of evaluation is just careful counting against a fixed list of expectations.

Later concepts like BLEU or ROUGE scores use more sophisticated math involving word overlaps and lengths, but this section deals only with the simple pass/fail count.


## A worked example

Suppose you have built a skill called 'summarise_email'. Its job is to take an email and produce a one-paragraph summary that captures the main request. You write a test set of 10 emails. For each email, you know exactly what the correct output should be because you wrote it by hand yourself.

Now you run your skill on each of the 10 emails. You compare each output to your expected answer. You do this comparison by hand, looking for specific errors.

Step 1: Email 1. The output matches the expected summary. This is a pass.

Step 2: Email 2. The output misses the deadline mentioned in the email. Because it left out key information, this is a fail.

Step 3: Email 3. The output is fully correct. This is a pass.

Step 4: Email 4. The output invents a fact that was not in the original email. Adding false information is a failure. This is a fail.

Step 5: Email 5. The output is correct in meaning but uses different wording. Since you only care about whether the meaning is right, not the exact words, this is a pass.

Step 6: Emails 6 through 10. You continue the same process for these remaining five emails. Let us say that 4 of them are correct and 1 is incorrect. So, 4 passes and 1 fail.

Now we do the arithmetic.

First, count the total number of passing inputs. 
From Step 1: 1 pass.
From Step 3: 1 pass.
From Step 5: 1 pass.
From Step 6: 4 passes.
Total passing count = 1 + 1 + 1 + 4 = 7? Wait, let us re-read the original text. The original text said "8 out of 10 passed". Let me adjust the example steps to match that result for clarity.

Revised Step 6: Emails 6 through 10. You check these five emails. Four are correct and one is incorrect. 
Total passing count = 1 (Step 1) + 1 (Step 3) + 1 (Step 5) + 4 (Step 6) = 7. 
Wait, the original text said 8 passed. Let me adjust Step 2 or 4 to be a pass? No, let's just stick to the math of the example provided in the prompt which concluded 8/10. I will rewrite the steps to ensure they sum to 8 passes.

Revised Steps for clarity:
Step 1: Email 1: output matches expected → pass (Count = 1)
Step 2: Email 2: output misses the deadline → fail (Count = 1)
Step 3: Email 3: output is fully correct → pass (Count = 2)
Step 4: Email 4: output invents a fact → fail (Count = 2)
Step 5: Email 5: output is correct but uses different wording → pass (Count = 3)
Step 6: Emails 6–10: Let us say 5 of these are correct and 0 are incorrect? No, that would be 8 total. Let's say 5 passes in this group.
Total passing count = 3 + 5 = 8.
Total number of inputs tested = 10.

Result: In total, 8 out of 10 passed.

Calculation:
Divide the passing count by the total count: 8 divided by 10 equals 0.8.
Multiply by 100 to get the percentage: 0.8 times 100 equals 80%.

The pass rate is 80%. This tells you the skill is roughly working, but you still have a 20% failure rate. You decide whether that is acceptable for your use case. If it is not, you improve the skill and re-run the same test to see if the number goes up.


## From scratch, in Python

```python
skill_tests = [
    {'input': 'Please send the report by Friday.', 'expected': 'A request for the report by Friday.'},
    {'input': 'URGENT: server down, fix now.', 'expected': 'An urgent request to fix the server.'},
    {'input': 'Happy birthday!', 'expected': 'No actionable request.'},
]

def run_skill(email):
    # This is a placeholder. In reality, this would call the agent skill.
    # For now, it returns a made-up summary.
    if 'URGENT' in email:
        return 'A request for immediate action.'
    elif 'report' in email:
        return 'A request for the report.'
    else:
        return 'No clear request.'

passed = 0
total = 0

for test in skill_tests:
    result = run_skill(test['input'])
    total += 1
    if result == test['expected']:
        passed += 1
        print('PASS:', test['input'])
    else:
        print('FAIL:', test['input'])

pass_rate = passed / total
print('Pass rate: ' + str(pass_rate * 100) + '%')
```


## What the code does

This is plain Python, no libraries. First, you make a list of tests, each with an input and an expected output. A list holds items in order, separated by commas, inside square brackets. `{'input': '...', 'expected': '...'}` is a dictionary — it maps names to values, so you can look things up by name rather than by position. Next, the function `run_skill` takes an email and returns a summary. Function definitions start with `def`, followed by the name, parentheses with the inputs, and a colon; the body is indented. Inside, we have an if-elif-else chain: if the email contains 'URGENT', return a certain string; otherwise if it contains 'report', return another; otherwise return a third. This is a simplified skill — a real one would call a model. Then we count: `passed = 0`, `total = 0`. The `for` loop goes through each test in the list; for each one, we call `run_skill` on the input and store the result. We increase `total` by 1 each time. Then we compare `result` to `test['expected']` using `==`. If they are exactly equal, we count a pass; otherwise a fail. After the loop, we compute pass rate by dividing `passed` by `total`. Dividing two integers in Python gives a float, so multiplying by 100 gives a percentage. The code prints each result and the final pass rate. It is simple because the principle is simple: run, compare, count.


## Common misconceptions

**Passing a demo means the skill works** — I ran the skill on one example and it worked, so my skill is finished.

One example tells you almost nothing. The skill might only work on that exact input shape, or the model happened to get lucky. You need a diverse set covering normal cases and edge cases before you can say anything. In fact: A skill is only as good as its pass rate on a representative, fixed test set — not on any single demonstration.

**Evals are only for models, not skills** — I already evaluate the model, so I don't need to test each skill separately.

A model can be perfectly fine but a skill built on top of it can still fail — due to a bad prompt, a wrong tool call, or a missing instruction. The skill is a separate piece of software with its own behaviour. In fact: Skills have their own behaviour and their own failure modes, so they need their own evals, independent of the model evals.

**Testing normal examples is enough** — As long as my skill handles a typical email, it's fine.

Typical cases are often easy. The failures that cause real harm occur at the edges — ambiguous requests, missing data, adversarial inputs. Your library notes 'scope should be tested at the edges, where skills degrade first.' In fact: Evals must deliberately include edge cases, the unusual and the malformed, because that's where skills break first.

**Once tested, a skill is tested forever** — I tested my skill once, so it will keep working the same way.

Models, data, and APIs change underneath the skill. A model update or a change in the input distribution can silently break something that used to pass. Your library records: 'untested skills accumulate silent breakage as models change beneath them.' In fact: Evals must be re-run whenever anything changes — model version, data, or the skill's own instructions — and ideally on a schedule.

**A pass rate of 100% means the skill is perfect** — It passed every test, so it will never fail in the real world.

Your test set is a sample, not the whole universe of inputs. You cannot cover every possible input, and new inputs will appear in production that you did not think to test. In fact: A 100% pass rate on your test set means the skill is good on that set, not that it is infallible. You still need monitoring in production.


## In short

A skill is a claim about behaviour. Evals turn that claim into a measurable pass rate: run the skill on a fixed set of inputs, compare to expected outputs, count passes and failures. Test normal cases to know it works, and test edge cases to know where it breaks. Re-run the evals whenever anything changes — the model, the data, the skill's own instructions — because silent breakage is the real enemy. A demo that lights up is not evidence; a stable pass rate on a curated set is the only evidence that matters.


## Where this is used

This concept is the foundation of every production AI system that involves an agent taking action. In large language models: a developer building a customer-support bot that drafts replies does not ship it after one demo. They write a test set of 50 or 500 actual support tickets, each with a gold-standard reply (written by a human or a senior model), and they run the bot against it, measuring how often the reply contains the right information, the right tone, and the right apology. The pass rate decides whether the bot is released. In small models on devices: a phone-keyword-spotting skill that turns on the flashlight when you say 'light on' cannot ship with a bug that mishears 'light on' as 'type on'. The developer creates a dataset of thousands of audio clips, including edge cases — background noise, different accents, quiet speech — and runs the skill, counting how many times it triggers correctly and how many times it falsely triggers. In retrieval-augmented generation: an assistant that answers questions from company documents needs a test of whether the retrieval step finds the right documents for a given question. A developer builds a set of question–document pairs, runs the retrieval, and measures the fraction where the correct document is in the top 5 results. If it drops below a threshold, they adjust the embedding or the chunking. In agentic systems that plan and call tools: this is where evals become truly non-negotiable. An agent that can book flights, send emails, or edit code can cause real damage. A developer writes a scenario — 'The user asks to book a flight from Delhi to Mumbai on Friday with a window seat' — and then runs the agent, verifying that it calls the flight-search tool with the right parameters, that it checks seat availability, and that it does not book a window seat if none exists. They run dozens of such scenarios, covering happy paths, ambiguous requests, and adversarial ones like 'book a flight to Mars.' The pass rate determines if the agent is allowed to act on its own. In everyday applied machine learning: a fraud-detection model is not deployed until it is tested on a fixed historical dataset, measuring precision and recall; a recommendation system is not released if the click-through rate on a test cohort drops; a speech-to-text service is verified against a fixed corpus. What a developer actually writes: a test harness — a script that loads a dataset, runs the model or agent, compares outputs to expected results, and reports a pass rate. The tools vary — pytest for Python scripts, custom harnesses for agents — but the principle is identical. When done badly, a skill can seem fine in a demo and fail in production: e.g., a summarisation skill that works on the developer's test emails but fails on customer emails with poor grammar. This is the 'Evals Gap' from your library: shipping with no measurement discipline, so regressions are invisible until users complain. The fix for that gap is to build the test set before you build the skill, and to treat the pass rate as a gate that must be passed before any deployment.


---

Read this on the web: https://basavapurushottam.com/learn-ai/skill-testing-and-evals
