# Agentic Test Authoring

Using agents to generate, maintain and repair test suites, driven from the application under test.

**Rests on:** Agent Executors

*AI Agents · AI Coding Assistants*


## Why learn this

This concept unlocks the next stage of your journey: how AI coding assistants go from suggesting snippets of code to autonomously working on a whole repository. Once you see how an agent writes and maintains tests for an application, you'll understand the broader pattern of agentic systems — the same machinery that powers agents that plan, call tools, and make multi-file changes. You'll also be ready to study the topics your library lists as neighbours: Agent Instruction Files, Skill Design and Scoping, and Claude Skills, because each of those is a way of giving an agent the context and the boundaries it needs to do test authoring well. Without this concept, those topics would feel like abstract advice; with it, they become concrete tools you can reach for.


## The intuition

Imagine you've just built a small web app — a to-do list where users can add, tick off, and delete items. Before you show it to anyone, you want to be sure it works. You could click through it yourself, trying every button, entering every kind of input you can think of: a normal task, an empty task, a task with a thousand characters, a task with special symbols. That's manual testing — a human, exploring by hand. Now imagine you could hire a tireless assistant who does exactly that clicking and typing for you, but doesn't get bored, doesn't miss a case, and does it a hundred times faster. That assistant is an agent — a specific type of program that can take actions in the world rather than just chat with you. This program is powered by a large language model (LLM), which is a software system trained on vast amounts of text to understand and generate human language, allowing it to interpret screenshots and code as if they were instructions or stories. Agentic test authoring is the practice of pointing such an agent at your app and letting it generate, update, and fix the automated tests that verify your app's behaviour. Automated tests are self-running scripts — sets of instructions that execute themselves without human intervention to check if the software behaves correctly. The agent explores the app the way a curious tester would, but it writes down what it finds as test code that runs automatically from then on. The key phrase is 'driven from the application under test' — the agent doesn't imagine tests from a spec alone; it actually interacts with the running app and sees the real screens, the real error messages, the real behaviour. That's what makes it powerful and also what makes it need human oversight, because the agent can only test what it can see and can encode its own wrong assumptions into the tests.


## An analogy

Think of a new cook joining a restaurant kitchen. The head chef doesn't hand them a recipe book and say 'go cook'—they set them loose in the actual kitchen, with real ingredients, real pans, and a real stove, and say 'make me a dish that shows you understand how this kitchen works.' The cook explores: they open the fridge, see what's there, try a combination, taste it, adjust. Their first attempt might be okay but not great; they try again, changing the proportions, until they have something the head chef approves. Then they write down the recipe so it can be repeated. Agentic test authoring is like that: the agent is the cook, the application is the kitchen, and the test suite is the written recipe. The agent 'tastes' by running the app, trying actions, and checking what comes back. It writes down what it learns as automated tests. But here's where the analogy stops: a cook understands flavour and can improvise when an ingredient is missing, but the agent only knows what it has seen in the app. If the app has a button that's hidden unless you scroll, the agent might never find it, so it won't test that path. And the recipe the agent writes — the tests — encodes its interpretation of the kitchen. If the agent saw a field labelled 'name' and assumed it required text, it will write a test that expects that, and if the real requirement was that empty names are allowed, the test is wrong. So the head chef (you) must review the recipe before it goes into the binder. The agent is a fast, thorough explorer, but you are the one who knows what the dish is supposed to taste like.


## Definition

Agentic test authoring is the practice of using AI agents — programs that can independently interact with an application, observe its behaviour, and modify code — to automatically generate, maintain, and repair the automated test suite for that application, with the agent's decisions being driven by what it actually encounters in the running app.


## How it connects

You've noted in your library that this sits under AI Coding Assistants. The parent concept's key idea is that we're moving from chat-based assistance (you ask, the model replies) to agentic execution — where the assistant acts on a real repository. Agentic test authoring is one concrete form of that execution. It builds directly on Large Language Models (the agent's brain) and AI Agents (the loop of plan-act-observe). You've also noted neighbouring topics: Agent Instruction Files and Skill Design and Scoping are exactly the tools you'd use to shape how a test-authoring agent behaves — they give the agent standing context about the codebase and define its boundaries. Claude Skills, another neighbour, are how you'd package a reusable test-authoring workflow so you can invoke it any time you start a new project.


## From scratch, in Python

```python
# Plain Python to simulate what an agent does when it explores an app.
# We'll pretend the 'app' is a simple function that returns user info.

def fake_app(user_id):
    """A stand-in for a real web app. Returns a string."""
    if user_id == 1:
        return "Alice, age 30"
    if user_id == 2:
        return "Bob, age 25"
    return "unknown user"

# The 'agent' is a function that tries different inputs and records what happens.
# First, let's write a tiny test that checks the app works as expected.

def test_app():
    # A test is a piece of code that checks a claim.
    # We use 'assert' to say 'this must be true'.
    result = fake_app(1)
    assert result == "Alice, age 30", f"Expected Alice, got {result}"
    print("Test passed for user 1")

# Run the test
test_app()

# Now, imagine the agent is exploring. It tries a user id it hasn't seen.
# It observes what the app returns, and writes a new test based on that observation.

def agent_explores():
    # The agent 'taps' the app with a new input.
    observed = fake_app(3)
    # It records what happened.
    print(f"Agent observed: {observed}")
    # It writes a test that checks the observed behaviour is preserved.
    # (In reality, the agent would write this into a test file.)
    def new_test():
        assert fake_app(3) == "unknown user"
        print("New test passes")
    return new_test

new_test = agent_explores()
new_test()
```


## The same thing, with a framework

```python
In practice, you wouldn't write a test agent from scratch. You'd use a framework like Pytest for writing the tests themselves, plus an agent framework such as LangChain or a tool like Claude Code that can interact with a browser or API. For example, a tool like Playwright (for browser testing) lets an agent control a real browser, click buttons, and inspect the page. The agent's 'observation' is the HTML it sees, and the 'test' it writes is a Pytest or Playwright test. The library call `page.click('button')` is the agent's finger, and `assert page.title() == 'Home'` is its assertion. Underneath, that library call is doing what our `fake_app` call did: sending an input and getting an output.
```


## What the code does

The code defines `fake_app`, a stand-in for a real application. It takes a user ID and returns a string. A test, `test_app`, checks that the app returns the right thing for user 1. It uses `assert`, which is Python's way of saying 'this must be true — if not, stop and raise an error.' Then `agent_explores` simulates the agent: it tries a new input (user 3), observes the output, and writes a new test function based on that observation. In real life, the agent would add that test to a suite of tests that run automatically. The key point: the agent's test is only as good as its observation. If the app returned 'unknown user' because it was a temporary glitch, the agent would write a test that bakes in the glitch. That's why review matters.


## Common misconceptions

**The agent's tests are correct because the agent saw the app** — The agent interacted with the app and observed what happened, so the tests it writes must be right.

The agent only sees what it explores. It might miss a condition that only occurs under certain inputs, or it might observe a bug and encode it as expected behaviour. In fact: The agent's tests are a proposal. They reflect what the agent saw, which may be incomplete or biased by the agent's assumptions. Human review is essential to catch cases the agent didn't explore and to reject tests that bake in incorrect behaviour.

**Generated tests replace manual testing** — If an agent writes tests, I don't need to do manual exploratory testing anymore.

Automated tests, whether written by humans or agents, only check what they check. Manual exploratory testing finds issues that automated tests miss — usability problems, visual glitches, unexpected user behaviour. In fact: Agentic test authoring complements manual exploratory testing. The agent handles the repetitive, exhaustive part; the human uses judgement and creativity to find things the automated suite (and the agent) would never think to test.

**The agent explores every possible case** — Because the agent can try many things quickly, it will cover all the edge cases.

The agent's exploration is driven by the paths it takes. It has no inherent understanding of what 'every possible case' means — it might not try an empty input, or a negative number, or a special character, unless it stumbles on them. In fact: The agent is a fast but blind explorer. It needs guidance — the test author (you) must specify the kinds of cases that matter, often via instruction files or seed tests, so the agent doesn't wander only where the app's visible behaviour leads it.

**The agent knows the requirements** — The agent sees the app, so it knows what the app is supposed to do.

The agent sees what the app does, not what it should do. If the app has a bug, the agent observes the buggy behaviour and writes a test that expects that bug. In fact: The agent's tests are based on observed behaviour, not specification. To write tests that check the true requirements, you must supply the requirements — either explicitly in the prompt or by having the agent compare against a known-good version.


## In short

Agentic test authoring is using AI agents to explore an application, observe its behaviour, and write automated tests from those observations. It is powerful because it can generate cases humans wouldn't think to script, and it is dangerous because the agent's tests encode its assumptions and observations, which may be wrong. The practice always requires human review: generated tests are a proposal, not a result. It complements, rather than replaces, manual exploratory testing. Master this and you understand the core of how AI coding assistants work — the loop of acting, observing, and learning that powers all agentic systems.


## Where this is used

Agentic test authoring is not a hypothetical — it is actively used across the AI development stack today. In **large language model development**, teams use agents to generate tests for the model's own evaluation harnesses. An agent given a set of prompt templates will try variations (different phrasings, different edge cases) and write tests that check the model's responses meet certain criteria — for example, that it never outputs harmful content, or that it always includes a required disclaimer. The mechanism: the agent sends prompts to the model, captures the outputs, and writes assertion-style tests that fail if a future model version returns something different. Developers then run these tests as part of the model's regression suite. In **small models running on devices**, agentic test authoring helps verify behaviour under resource constraints. An agent can be pointed at a TinyML model's inference code and asked to generate tests that run on a simulated device, checking memory usage or latency. The mechanism: the agent reads the code, writes unit tests that measure execution time, and adjusts them as the code changes. In **retrieval-augmented generation (RAG)**, the challenge is that the quality of answers depends on what's retrieved. An agent can generate tests that verify: given a query, the retriever returns the right document, and the generator's final answer is grounded in that document. The agent does this by constructing a set of test queries, running them through the RAG pipeline, and asserting on the retrieved results. In **agentic systems themselves**, the tests are even more crucial because the agent is non-deterministic. An agentic test authoring agent (meta, but real) can be used to generate tests for another agent's planning loop — feeding it synthetic user requests and asserting it calls the right tools in the right order. The mechanism: the testing agent observes the tool calls the planning agent makes and writes tests that encode the expected sequence. Finally, in **everyday applied machine learning**, where you're building a recommendation system or a fraud detector, agentic test authoring is used to generate tests for data preprocessing code — the agent feeds in various data formats (including dirty data) and asserts the preprocessing doesn't crash. What a developer actually writes: they configure an agent (often via a CLI or a persistent instruction file) to target a specific folder of code, run the suite, and review the agent's changes in a pull request. The quality gate is always human review — the agent's tests are a proposal, not law. If done badly — if you let the agent write tests without review — you end up with a suite that passes but tests nothing meaningful, or worse, encodes the agent's hallucinations as expected behaviour.


---

Read this on the web: https://basavapurushottam.com/learn-ai/agentic-test-authoring
