# Gold and Adversarial Sets

Gold sets capture correct expected behaviour; adversarial sets deliberately probe for failure.

**Rests on:** Evaluation Datasets

**Leads to:** Adversarial Examples, MITRE ATLAS

*LLMs and Generative AI · LLM Evaluation*


## Why learn this

You are building an AI assistant that makes promises to users. If it gives bad advice, your product suffers. You need a reliable way to catch those mistakes before they reach customers. Right now, you might have a vague sense of whether answers are 'good' or 'bad,' but that is not enough for a professional tool. To fix this, you need two specific lists: one containing questions with perfectly known correct answers (the gold set) and another containing tricky traps designed to catch the model making errors it usually hides (the adversarial set). Using these two sets together lets you turn vague feelings into concrete numbers. For example, instead of guessing if an update helped, you can see that the accuracy on your test cases dropped from 95% to 88%, which tells you immediately not to release that update.


## The intuition

Imagine you are a teacher preparing for exams. To know if your students have truly learned the material, you cannot just ask them questions they have seen before in class. You need two distinct sets of questions.

First, create the 'gold set'. This is a collection of standard problems where you already know the exact right answer. For instance, if you are testing a calculator app, your gold set might include: 'What is 2 + 2?' (Answer: 4), 'What is 10 times 5?' (Answer: 50). These questions prove the system works for normal, everyday tasks. If the calculator says '2 + 2 = 5', it fails this set. This set establishes a baseline of trust.

Second, create the 'adversarial set'. A good teacher knows students might memorize answers without understanding or fall for tricks. So, you add questions designed to expose those specific weaknesses. If you suspect students confuse similar words, you ask: 'What is the plural of sheep?' (Answer: sheep, not sheeps). If you suspect they struggle with ambiguity, you ask: 'Is the bank open?' without saying which river or financial institution. These questions are not random; they are targeted traps. They test if the system breaks under pressure or confusion.

Why do you need both? If you only use the gold set, you might pass all tests but still fail when a user asks a slightly unusual question that looks easy but requires careful thought. If you only use the adversarial set, you might learn where it breaks but not know if it can handle basic requests reliably. You need the gold set to confirm competence and the adversarial set to find hidden cracks.


## An analogy

Think of testing a new car model before it hits the road. You do not just drive it on a smooth, empty highway to see if the engine turns on; that is your 'gold set' test—it confirms the basic function works. Instead, you also take it to a specialized skid pad with slippery oil patches and sharp turns at high speeds; this is your 'adversarial set'. You deliberately create conditions where loss of control is likely, not because you expect the car to fail every time, but because you need to know exactly how it handles instability. Does the anti-lock braking system activate correctly? Do the tires hold up? By subjecting the car to these controlled, high-stress scenarios, you learn about its limits and safety features in a way that normal driving never reveals. Similarly, for an AI model, the gold set confirms it can drive the car, while the adversarial set tells you what happens when it starts to skid.


## Definition

A gold set is a collection of test cases with known correct answers used to verify standard performance, while an adversarial set is a collection of carefully crafted edge-case inputs designed to reveal specific weaknesses or failure modes in the system.


## How it connects

Once you have these two sets established, you can apply various methods to score them. You can use answer relevancy checks to ensure the output stays on topic, and deterministic validators (automated rules that check for exact patterns, like ensuring a phone number format is correct) to catch obvious errors. You can compare outputs against a known good reference using metrics that calculate similarity, often referred to as BLEU scores in industry reports, which measure how closely the generated text matches the ideal answer. Finally, you can set up regression gates (automated stops that prevent any software update from being released if the new version scores lower on your gold set than the previous version did).


## The mathematics

This concept is about the structure of a dataset, not about arithmetic calculations. The mathematics involved is set theory, which is the branch of mathematics that deals with collections of things. But you don't need to know any set notation to understand the idea. A 'set' is just a collection of distinct items. In our case, each item is an input (like a question or a prompt) and its expected output (like the correct answer). A gold set is a set of such pairs. An adversarial set is also a set, but the inputs are chosen to be difficult or misleading. The key mathematical idea is that these two sets overlap – an input could be in both if it is both normal and tricky. But for evaluation, we usually keep them separate so we can measure two different things. When you evaluate a model, you run it on each input in the gold set and compare its output to the expected output. You count how many match (correct answers) and how many don't (errors). That gives you a score like accuracy: the fraction of inputs where the model got it right. For the adversarial set, you do the same, but you interpret a failure as revealing a weakness. There is no formula to derive, but understanding the logic of comparing outputs is central. To compare, you need a way to say whether two pieces of text are 'the same' – that is a measurement problem, but not a formula you need to write. In practice, you might use a simple exact match (the model's output string equals the expected string) or a more lenient method like BLEU, which counts overlapping words. But the core is that you are measuring the model's behaviour against a reference.

Let G be a gold set of (input, expected) pairs. Let A be an adversarial set. For each input i, the model outputs o_i. We define a comparison function match(o_i, expected_i) that returns 1 if they match and 0 otherwise. Then accuracy on the gold set is (sum of match over G) / (number of items in G). Similarly for A. This is a simple ratio, not a complex formula.


## A worked example

```
Let's build a tiny gold set for a chatbot that answers questions about solar panels. We have five inputs with expected answers (the gold set). Then we create an adversarial set of three tricky inputs.

Step 1: Gold set: (input: 'How much does a solar panel cost?', expected: 'It depends on the size and quality. A typical 300-watt panel costs around ₹8,000 to ₹15,000.')   # A normal question with a factual answer.
Step 2: Gold set: (input: 'What is the lifespan of solar panels?', expected: 'Most panels last 25–30 years with proper maintenance.')   # Another typical question.
Step 3: Gold set: (input: 'Do solar panels work on cloudy days?', expected: 'Yes, but at reduced efficiency.')   # A question that tests basic knowledge.
Step 4: Adversarial set: (input: 'Should I buy a solar panel for my apartment in Mumbai?', expected: 'That depends on your roof space and electricity needs.')   # This is tricky because it asks for a recommendation, not a fact. The model might hallucinate a specific price or advice.
Step 5: Adversarial set: (input: 'What is the warranty on a solar panel that never gets sunlight?', expected: 'A panel that never gets sunlight will not produce power, but the warranty covers defects, not lack of use.')   # This is a trick: it combines a nonsense condition with a legal question.

Result: We have two sets. When we test a model, we run it on each input and compare to the expected. Suppose the model answers 4 out of 5 gold correctly, and 1 out of 3 adversarial correctly. Gold accuracy = 4/5 = 80%. Adversarial correctness = 1/3 = 33%. The low adversarial score shows the model is weak on tricky or ambiguous inputs, even though it performs well on normal ones.
```


## From scratch, in Python

```python
# We'll build a small evaluation function in plain Python.
# First, we need a way to compare two texts. For simplicity, we use exact match.
# A list in Python is a collection of items in order. We'll put our gold set as a list of tuples.
# A tuple is like a list but cannot be changed. It holds two values: the input and the expected.

# Our gold set:
gold_set = [
    ("How much does a solar panel cost?", "It depends on the size and quality. A typical 300-watt panel costs around Rs 8,000 to Rs 15,000."),
    ("What is the lifespan of solar panels?", "Most panels last 25-30 years with proper maintenance."),
    ("Do solar panels work on cloudy days?", "Yes, but at reduced efficiency.")
]

# Our adversarial set:
adversarial_set = [
    ("Should I buy a solar panel for my apartment in Mumbai?", "That depends on your roof space and electricity needs."),
    ("What is the warranty on a solar panel that never gets sunlight?", "A panel that never gets sunlight will not produce power, but the warranty covers defects, not lack of use.")
]

# A function that checks exact match. A function is a block of code we can reuse.
def exact_match(output, expected):
    return output == expected

# A function to evaluate a model given a set of test cases.
def evaluate(model, test_set):
    correct = 0
    total = len(test_set)  # len gives the number of items in a list
    for input_text, expected in test_set:
        # 'for' loop goes through each item in the list
        output = model(input_text)  # call the model function
        if exact_match(output, expected):  # if they are exactly equal
            correct = correct + 1
    return correct / total  # accuracy as a fraction

# A dummy model that just returns a fixed string for demonstration.
def dummy_model(input_text):
    # For the first adversarial input, return a wrong answer
    if input_text.startswith("Should"):
        return "I don't know, but I'd say yes."
    # Otherwise, return the first expected (not correct for all)
    return "It depends on the size and quality."

# Now run evaluation:
gold_accuracy = evaluate(dummy_model, gold_set)
adv_accuracy = evaluate(dummy_model, adversarial_set)

print("Gold accuracy:", gold_accuracy)  # likely 1/3 = 0.333
print("Adversarial accuracy:", adv_accuracy)  # likely 0/2 = 0.0
```


## What the code does

We start with two lists of tuples. A list holds items in order; a tuple holds two values that belong together. The first tuple is (input, expected). We then define a function exact_match that takes two strings and returns True if they are exactly the same, otherwise False. The evaluate function takes a model (which is also a function) and a test set (a list of tuples). It loops over each pair, calls the model with the input, and compares the output to the expected using exact_match. It counts how many are correct and divides by the total. dummy_model is a fake function that returns a wrong answer for questions starting with 'Should', else a generic answer. When we run evaluate on the gold set, only the first gold input gets the correct expected answer (the one starting with 'How'), so it gets 1 out of 3. The adversarial set: two inputs, one starts with 'Should', the other starts with 'What'. For the second, dummy_model returns 'It depends...', which does not match the expected, so 0 out of 2. The output shows those fractions. This simple script teaches you the core logic: you have a list of expected pairs, a function to compare, and a loop that computes accuracy.


## Common misconceptions

**Adversarial means random or weird** — Adversarial sets are just random, weird questions you can think of to trip up the model.

If you just make up random nonsense, you might test things the model was never designed to handle, and the failures don't tell you about real weaknesses in your product. A good adversarial set targets specific failure modes that are likely to occur in real use, based on incidents or error analysis. In fact: Adversarial sets are carefully designed to stress test particular capabilities – they probe for known or likely weaknesses, not just general absurdity.

**One set is enough** — If the model passes my gold set, I don't need an adversarial set.

Passing the gold set only shows that the model can handle normal cases. It says nothing about how it behaves on tricky or edge-case inputs. The adversarial set catches those failures, so both are necessary. In fact: Gold sets measure typical performance; adversarial sets measure robustness. You need both to understand where the model works and where it breaks.


## In short

Gold sets measure whether a model produces the correct output for typical inputs; adversarial sets probe for failures on tricky or edge-case inputs. Both are essential for evaluating an LLM application. Gold sets tell you if it works; adversarial sets tell you where it breaks.


## Where this is used

In real-world LLM applications, gold and adversarial sets are the heart of evaluation. For a large language model like GPT-4, a gold set might be a collection of prompts with human-written ideal answers. For example, a company building a customer support chatbot creates a gold set of 1,000 typical customer queries with the correct responses, so they can measure whether each new model version is as good as the old one. If the new model's accuracy on the gold set drops from 95% to 90%, they know it has regressed. Adversarial sets are used to test safety and robustness. For a model that generates code, an adversarial set might include prompts like 'Write a function that deletes all files' or 'Explain how to make a bomb' – these are designed to see if the model has harmful behaviours. Companies like OpenAI and Anthropic release 'red-teaming' datasets that are adversarial. In retrieval-augmented generation (RAG), where a model answers questions by pulling from a database, an adversarial set might include questions that are not in the database but look like they should be, to see if the model hallucinates. For an agentic system that calls tools (like a calculator or a search engine), an adversarial set might include prompts that ask the agent to perform an action that is not possible, or that are ambiguous, to test if it correctly asks for clarification. In small models running on phones, an adversarial set might include noisy speech inputs or unusual accents to make sure the model still works. In applied machine learning for fraud detection, a gold set might be historical transactions labelled as fraud or not, and an adversarial set might be new types of fraud that have never been seen – you deliberately include examples that look legitimate but are not, to see if the model catches them. The key is that an adversarial set must come from real failure modes you have observed, not just imagination – as your library notes remind you, 'adversarial cases should grow from real incidents'.


---

Read this on the web: https://basavapurushottam.com/learn-ai/gold-and-adversarial-sets
