# Prompt Iteration

Refining a prompt through cycles of change and measurement rather than intuition.

**Rests on:** LLM Nondeterminism

**Leads to:** Prompt Optimization, Prompt Registry and Versioning, Trace-First Development

*LLMs and Generative AI · Prompt Engineering*


## Why learn this

This skill turns writing instructions for an AI from guessing into a reliable process. When you finish this section, you will be able to take any request — a product description, a customer-support script, or a coding task — and improve it systematically. You will learn to change one specific detail, test the result, measure the outcome, and repeat, just as a developer fixes software bugs. You will move from saying 'this feels better' to saying 'this version works because...' with proof. This discipline is essential for two practical tools you will encounter later: using a program to automatically find the best instructions (automated search) and keeping a structured history of your drafts so you can undo mistakes without losing progress. Without this method, those tools have no foundation.


## The intuition

Think of the last time you asked someone for a favor and they misunderstood you. You likely did not just repeat the request louder. You reworded it: 'No, I meant the blue one, not the red one.' You changed one specific detail, checked their response, and adjusted again. This is iteration. Prompt iteration is this same disciplined process applied to an AI model.

Here is how it works in practice:
1. You start with a baseline prompt.
2. You test it on a collection of questions where you already know the correct answers (this collection is called an **evaluation set**, which is simply a test bank of known good examples).
3. You measure how many answers the AI got right.
4. You change exactly one thing in the prompt — one word, one example sentence, or one formatting rule.
5. You run that same evaluation set again and compare the new score to the old one.

The critical rule is changing only one variable at a time. If you change three things simultaneously and the performance improves, you do not know which of the three changes helped. If it gets worse, you do not know which one caused the damage. By isolating each change, you can attribute the result directly to that specific adjustment. This is the scientific method applied to writing prompts. Without this measurement step, you are guessing. With it, you build reliable results.


## An analogy

Imagine you are a chef creating a new dish. You have a base recipe and want it to be perfect. You do not throw random ingredients in and hope for the best. You start with a standard version, cook it, taste it, and score it out of ten — let’s say it gets a 6.

Then you change one thing: you add a pinch of salt. You cook it again, taste it, and score it. Now it is a 7. You then change another single thing: you reduce the cooking time by two minutes. Now it scores an 8.

If you had added salt and reduced the time at the same time, you would not know which adjustment improved the dish — perhaps one helped while the other hurt, canceling each other out. The same logic applies to prompts: you have a test bank of inputs, run your prompt against it, score it, change one word, and score it again.

Note that this analogy is imperfect because a recipe is fixed steps, whereas an AI interprets instructions differently depending on its version or context, which is why keeping the test bank constant is vital.


## Definition

Prompt iteration is the practice of improving an instruction set by altering exactly one element at a time and measuring the result against a fixed collection of known examples.

Formally, it is a cycle of modification and evaluation designed to isolate the cause-and-effect relationship between specific prompt changes and output quality.


## How it connects

This concept supports **Prompt Optimization**, which is the automated process where a computer program performs this cycle of changing and testing instructions for you, and **Prompt Registry and Versioning**, which is the system of saving each version of your prompts so you can track progress or revert to a previous working state if a change fails. It also underpins the later topic of **RAG** (Retrieval-Augmented Generation), where the information retrieved to support the prompt changes, making this iterative testing even more critical.


## The mathematics

This concept rests on a simple branch of mathematics: arithmetic — counting and comparing numbers. You will not need calculus, algebra, or any advanced symbols. The only operations you need are addition, subtraction, division, and comparison.

The core idea is to measure the quality of a prompt with a single number called a score. We calculate this score by looking at how many times the prompt gives a correct answer compared to how many questions we asked in total. This is just a fraction, which we then turn into a percentage.

Here is the step-by-step process for the mathematics:

1. Count the total number of test questions you have. Let us call this number 'total'.
2. Run your prompt on these questions and count how many answers are correct. Let us call this number 'correct'.
3. Divide the number of correct answers by the total number of questions. This gives you a decimal.
4. Multiply that decimal by 100 to get the score as a percentage.

The formula looks like this:
score = (correct divided by total) multiplied by 100

When you change your prompt, you repeat these steps. You get a new score. To see if the change helped, you subtract the old score from the new score. This difference is called the improvement.

improvement = new_score minus old_score

You then look at the sign of this number:
- If improvement is greater than 0 (positive), the new prompt is better. You keep the change.
- If improvement is less than 0 (negative), the new prompt is worse. You revert to the old prompt.
- If improvement equals 0, the change made no difference. You can choose to keep it or discard it based on other factors, such as whether the new prompt is shorter or faster.

Why is this arithmetic sufficient? Because the fundamental question in prompt iteration is not 'what is the absolute truth?' but rather 'is this version better than the previous version?'. This is a comparison of two counts. No complex equations are needed to answer that question.

Symbol glossary:
- score: The percentage of correct answers, a number between 0 and 100.
- correct: The count of queries where the prompt's answer matched the ideal response.
- total: The total number of queries in your evaluation set.
- improvement: The numerical difference between the new score and the old score. It tells you the direction and magnitude of change.


## A worked example

You have an evaluation set of 10 customer-support queries. For each query, you know the ideal response (what a perfect answer would say). You start with a current prompt template: 'You are a helpful assistant. Answer the customer's question.'

Step 1: Establish the baseline.
Run the current prompt on all 10 queries. Count how many answers are correct. Suppose you get 7 correct answers.
Calculation:
score_old = (7 divided by 10) multiplied by 100
score_old = 0.7 multiplied by 100
score_old = 70%
The baseline score is 70%.

Step 2: Make one change.
Change the prompt to: 'You are a helpful assistant. Answer the customer's question concisely, in under 50 words.' This is the only change you made.

Step 3: Measure the new result.
Run this new prompt on the same 10 queries. Count how many answers are correct. Suppose you get 8 correct answers.
Calculation:
score_new = (8 divided by 10) multiplied by 100
score_new = 0.8 multiplied by 100
score_new = 80%
The new score is 80%.

Step 4: Compare the scores.
Calculate the improvement by subtracting the old score from the new score.
improvement = 80 minus 70
improvement = +10
Because the improvement is positive (+10), the change was beneficial. You keep this new prompt version.

Step 5: Test another change.
Now, modify the prompt again to: 'You are a helpful assistant. Answer the customer's question concisely, in under 50 words, and always start with 'Dear customer'.' Run it on the same 10 queries. Suppose you get only 6 correct answers.
Calculation:
score_newer = (6 divided by 10) multiplied by 100
score_newer = 60%
Compare this to the previous best score (80%).
improvement = 60 minus 80
improvement = -20
Because the improvement is negative (-20), this change hurt the performance. You revert to the prompt from Step 3.

Result: After several cycles of making one small change, measuring the score, and comparing it to the previous best, you arrive at a prompt that scores 9 out of 10 (90%). You know exactly which changes helped and which didn't because you isolated each variable and used simple arithmetic to track the progress.


## From scratch, in Python

```python
# Prompt iteration - the core loop
# We'll simulate a tiny evaluation set and a prompt that we improve.

# Step 1: Build an evaluation set.
# A list of pairs: (query, ideal_answer)
# A list is a collection of items in order, written in square brackets.
# Each item here is a pair (a tuple), written in parentheses, that cannot be changed.

eval_set = [
    ("What is your return policy?", "Returns are accepted within 30 days."),
    ("How do I track my order?", "Go to the order confirmation email."),
    ("Do you ship internationally?", "Yes, to over 40 countries."),
    ("What is your phone number?", "+91-800-555-0100."),
    ("Can I change my delivery address?", "Yes, before dispatch."),
    ("What if my product is damaged?", "We send a replacement free of charge."),
    ("Do you offer gift wrapping?", "Yes, at checkout."),
    ("How long does delivery take?", "5-7 business days."),
    ("Can I cancel my order?", "Yes, within 24 hours."),
    ("What payment methods do you accept?", "UPI, credit card, and net banking.")
]

# Step 2: Define a function that runs a prompt on the eval set and returns a score.
# A function is a reusable block of code. We name it, give it an input (the prompt), and it returns an output (the score).

def score_prompt(prompt):
    """Return the number of queries the prompt answers correctly."""
    # In a real system, you would send the prompt to the model.
    # Here we simulate the model's answer by checking if the prompt contains the keyword.
    # This is a stand-in for actual model responses.
    correct = 0
    for query, ideal in eval_set:
        # The model's simulated answer is: the ideal answer, but we pretend the prompt affects it.
        # For this demo, the model 'succeeds' if the prompt mentions a relevant keyword.
        # This is not how real LLMs work, but it lets us see the iteration mechanics.
        if "concise" in prompt and "answer" in prompt:
            # Assume that this prompt makes the model answer correctly for all queries.
            correct += 1
        else:
            correct += 1  # In this fake world, all prompts score 10. This won't show improvement, so let's make it realistic.
    # Actually, let's make the scoring depend on the prompt in a meaningful way.
    # We'll use a simple heuristic: the prompt must contain certain keywords to be good.
    # This is a toy, but the mechanics are what matter.
    correct = 0
    for query, ideal in eval_set:
        # The model's answer is simulated: if the prompt has 'helpful' and 'concise', the model answers correctly.
        if "helpful" in prompt and "concise" in prompt:
            correct += 1
        else:
            # Otherwise, the model answers correctly only for some queries (e.g., those with '?')
            if query.endswith('?'):
                correct += 1
    return correct

# Step 3: Start with a baseline prompt.
prompt = "You are an assistant."
print("Baseline score:", score_prompt(prompt), "/", len(eval_set))

# Step 4: Iterate - change one thing at a time and measure.
# We'll keep a list of prompts to try.

candidates = [
    "You are an assistant.",
    "You are a helpful assistant.",
    "You are a helpful assistant. Answer concisely.",
    "You are a helpful assistant. Answer concisely and politely."
]

best_prompt = prompt
best_score = score_prompt(prompt)

for candidate in candidates:
    s = score_prompt(candidate)
    print(f"Prompt: {candidate} -> score: {s}/{len(eval_set)}")
    if s > best_score:
        best_score = s
        best_prompt = candidate

print("Best prompt:", best_prompt, "with score", best_score, "/", len(eval_set))

```


## What the code does

Let's walk through the code. First, we define an evaluation set as a list of tuples. A list is an ordered collection in square brackets; a tuple is an unchangeable pair in parentheses. Each tuple has a query and the ideal answer. The `score_prompt` function takes a prompt string and returns a score. Inside, we have a loop: `for query, ideal in eval_set:` — this goes through each pair in the list, unpacking query and ideal. In a real system, you would call the model here, but we simulate it with a keyword check. The loop counts how many queries the prompt 'answers correctly' based on whether the prompt contains certain words. This is a toy simulation — the point is not the simulation but the loop structure: define a base, try variations, compare scores. Then we have a list of candidate prompts, and we loop through them, computing the score for each and keeping track of the best. This is exactly what you would do manually, but the computer does it for you. The `print` statements show the scores so you can see the effect of each change. In a real workflow, you would not simulate the model — you would call the actual model API. But the structure is the same: create an eval set, define a scoring function, iterate over prompts, compare scores.


## Common misconceptions

**Prompt iteration is just guessing until it works** — I just tweak the wording until the model gives me what I want; there's no method to it.

Guessing is random. Iteration is systematic: you change one thing, measure against a fixed test set, and use the result to decide the next change. Without measurement, you don't know if you improved or just got lucky. In fact: Prompt iteration uses a defined loop: write a baseline, test on a fixed evaluation set, change one variable, test again, compare. It turns trial-and-error into a controlled experiment.

**Changing multiple things at once is fine as long as it works better** — I changed the wording, added an example, and changed the format, and the response got better, so the new prompt is better.

When you change several variables, you cannot attribute the improvement to any one. What if the example helped but the format hurt, and they cancelled out? You would keep a prompt that could be even better with a smaller change. In fact: Change exactly one thing at a time. The improvement from a single change is attributable and useful. If you want to change multiple things, do it sequentially, testing after each.

**An eval set is optional if my prompt seems to work** — I don't need a test set; I can just look at a few outputs and see if it's good.

Your few examples are not representative. The prompt might work on those but fail on other valid inputs. Without a fixed eval set, you cannot measure progress over iterations — you are just comparing against memory. In fact: Always create a small set of representative inputs with known good outputs. Run every iteration against that set. It gives you a consistent yardstick to measure against.

**Iteration is only for the prompt text, not for the context or retrieval** — My prompt is the only thing I can change; the retrieval results are fixed.

The context you feed the model is part of what the model sees. Changing retrieval changes the meaning of the prompt. If you change both the prompt and the retrieval, you are changing two variables and lose the ability to attribute. In fact: Treat the prompt and the context as coupled. If you iterate on the prompt, keep the retrieval fixed. When you change retrieval, re-evaluate the prompt from scratch. One change at a time applies across the entire input.

**If the score goes up, the change is always good to keep** — The new prompt got 8 correct on the eval set, so it's better and I should use it.

A single set of 10 examples might give a misleading result. A change might improve on those 10 but hurt on other real-world inputs. Also, the eval set is a sample; better to use a larger set or multiple runs, and consider other criteria like response length, safety, or latency. In fact: Use a sufficiently large and representative eval set. Treat an improvement as a hypothesis to confirm with more data, and weigh other factors alongside the score, such as consistency, safety, and cost.


## In short

Prompt iteration is a controlled loop: define an evaluation set with known good outputs, score your current prompt against it, change one thing, score again, and keep the change only if it improves the score. Always change one variable at a time so you can attribute the result. Without an eval set, iteration is just churn. This is the foundation for prompt optimization (automating the loop) and for reliable prompt engineering in any LLM application.


## Where this is used

Prompt iteration is used everywhere in AI development. In **large language models**, developers rarely write a prompt once and ship it. They build an evaluation set — a collection of inputs with known good answers — and iterate on the prompt until it meets a quality bar. For example, a team building a chatbot for a bank might have a hundred sample customer queries and a list of ideal responses. They write a prompt, run it on all hundred, count how many match, and change the prompt based on failures. This is not a one-time step: every time the model version changes, the prompt may need re-iterating, because the model's behaviour shifts. In **small models running on devices** (like a wake-word detector or a phone assistant), prompt iteration is even more critical because the model is smaller and more sensitive to phrasing. Developers test on device with a fixed eval set and iterate until the accuracy meets the requirement. In **retrieval-augmented generation (RAG)**, prompt iteration is intertwined with retrieval. The prompt includes retrieved passages, and changing the retrieval changes what the prompt 'means' to the model. So a developer might iterate on the prompt while also tuning the retrieval — but the discipline of changing one thing at a time applies across both. For example, if you change the retrieval top-k from 3 to 5 and also change the prompt, you cannot tell which caused the improvement. In **agentic systems** that plan and call tools, the prompts that control the agent's behaviour are iterated on extensively. The agent's system prompt determines how it chooses tools, how it handles errors, and how it formats its output. Developers run the agent on a suite of tasks, measure success rate, and iterate on the prompt — each change tested against the full suite. In **everyday applied machine learning** — recommendation, vision, forecasting, fraud detection — prompt iteration shows up wherever a model is used via an API, especially for classification or extraction. A fraud-detection system might use a prompt to summarize transaction details; the prompt is iterated on with historical fraud cases to maximize recall. In terms of libraries and frameworks, prompt iteration is not tied to a specific library. A developer might use Python to call an LLM API, with a script that loops over prompt candidates and computes scores. Tools like OpenAI's Evals or LangChain's evaluation modules help structure this, but the core is the same: an eval set, a scoring function, a loop. What breaks when it is done badly is everything: if you change multiple things at once, you cannot learn; if you do not have an eval set, you are just guessing; if your eval set is not representative of real use, you will overfit to the synthetic data. The discipline of measuring before and after every change is what separates prompting as an art from prompting as an engineering practice.


---

Read this on the web: https://basavapurushottam.com/learn-ai/prompt-iteration
