# Test Planning

Deciding what to test, at which level and to what depth, given risk and available effort.

**Rests on:** Risk Tiers

**Leads to:** API Testing, Backtesting LLM Strategies, Exploratory Testing Frameworks, Pairwise Testing, Property-Based Testing, Security Testing, Shift-Left Testing

*Software Engineering · Software Testing*


## Why learn this

Every project starts with a choice: what to build and how to ensure it works. Without a plan, you risk spending weeks writing code that fails or doesn’t do what the user needs. A clear test plan turns this uncertainty into a roadmap. It lets you use Test-Driven Development, where you write the rules of success before you start coding, ensuring you know exactly when you’re done. It enables Continuous Integration and Deployment, meaning your work is checked automatically every time you save changes, so bugs are caught immediately rather than weeks later. It supports Code Quality and Refactoring, allowing you to improve your code’s structure without fear of breaking existing features, because your tests will alert you if anything breaks. In systems that learn from data, a test plan helps you decide whether to verify the smallest part (a single calculation), the connections between parts (data moving through the system), or the entire user experience (clicking through the app). This concept gives you the decision framework you will reuse in every subsequent testing topic.


## The intuition

Testing is about answering two questions: 'What could go wrong?' and 'How much effort is it worth to check it?' Imagine you're a quality inspector for a factory that makes cricket bats. You have a limited budget. You don't inspect every single bat with equal thoroughness — the junior bats for ₹500 get a quick look, but the international pro bats for ₹50,000 get a full lab test. Why? Because a defect in a cheap bat costs little, but a crack in an expensive bat could ruin a player's career and your reputation. The same logic applies to software. Some functions are rarely used and cheap to fix if they break; others are in the critical path of every user and cause outages or financial loss. Risk-driven test planning means you allocate your testing effort where a failure would hurt most. You also decide the level: a unit test checks one function in isolation (fast, cheap, but doesn't prove the whole system works), an integration test checks two components together (like the bat fitting the handle), and an end-to-end test checks the whole flow from user click to database response (slow, brittle, but confidence-building). The plan is not a list of every possible test — it's a strategy for where to put your effort to get the best safety per hour of work.


## An analogy

Think of test planning like planning a medical check-up. You have a limited appointment time, and you want to catch the most dangerous conditions first. The doctor doesn't run every possible test on everyone — she asks about symptoms, risk factors, and family history. A young athlete with a knee pain gets an MRI for the knee, not a full body scan. A person over 50 with a family history of heart disease gets an ECG and blood tests, not a skin allergy panel. The decision is risk-based: high likelihood of a dangerous condition + high cost of missing it = more testing. The level of testing is like choosing between a blood test (unit — checks one biomarker) and a full stress test (end-to-end — checks the whole cardiovascular system under load). The blood test is cheap to repeat and pinpoints a specific issue, but it won't tell you if the heart can handle a marathon. The stress test is expensive and takes time, but gives you confidence that the whole system works together. Just as a doctor's plan is a written strategy, not a random list of tests, a software test plan is a document that says: 'These are the risks, these are the levels, this is how much effort each gets.' Where the analogy breaks down: a medical check-up is done periodically on a healthy person, but software tests run continuously, every time code changes. Also, in software, the plan is not just for one check-up — it's a living document that the team updates as the system evolves. And unlike a doctor, you can automate tests to run thousands of times a day, so the plan is about initial design, not just budget allocation. The plan's value is that it forces you to think: 'Given my limited testing time, what must not break? What would hurt the most if it failed?'


## Definition

Test planning is the systematic process of deciding what to test, at which level (unit, integration, end-to-end), and to what depth, based on the risk of failure and the cost of that failure, balanced against the effort available for testing.


## How it connects

You have no prior notes in this library yet, so we are starting fresh. This concept belongs to Software Testing, and it connects directly to your other notes: it leads naturally into Test-Driven Development (where you plan a test *before* writing the code) and Continuous Integration and Deployment (where your planned tests run automatically). The test plan decides the shape of the automation you will later build with tools like Playwright and Selenium for end-to-end tests, and it uses the systematisation tools of Equivalence Partitioning and Boundary Value Analysis to choose *which* cases to include. Contract Testing is a level of testing you will choose in the plan when you have separate services. The plan also decides whether you need Accessibility Testing or Performance Testing as separate tracks, based on the risk profile of your application. In the world of AI, a test plan for an ML system is even more crucial because the model's behaviour is probabilistic — you cannot just check 'does it work', you must define acceptable error rates and test across different inputs (see 'Application' below).


## The mathematics

This concept does not rest on heavy mathematics — it is about decision-making under constraints. However, the underlying branch is probability and risk assessment, because you are estimating the likelihood of a failure and the cost of that failure. You will use simple arithmetic: multiplying likelihood by cost to get a risk score, and adding up the effort for planned tests to see if it fits your budget. The key operation is a weighted sum, where each risk is weighted by its importance. You do this mentally all the time: 'If the server crashes, that's bad (cost=100), and it happens rarely (likelihood=0.01) — so the risk score is 1. If the login fails for one in ten users (cost=50, likelihood=0.1), the risk score is 5 — so I should test login more than the crash scenario.' That is it. No calculus, no algebra beyond multiplication and addition. The 'formula' you see below is just a compact way of writing this everyday arithmetic. The deeper mathematics of testing — like determining statistical confidence for random testing — is beyond this beginner page, but we will not need it.

risk_score = likelihood × cost
allocation_fraction = risk_score ÷ total_risk_score
effort_to_feature = allocation_fraction × total_effort


## A worked example

You are planning tests for a mobile banking app. You have identified four features that could fail. For each, you estimate a likelihood (between 0 and 1, where 1 means it will surely fail) and a cost (on a scale of 1 to 100, where 100 is a catastrophic loss of customers or money). Your total available testing effort is 20 person-hours.

Step 1: Login: likelihood = 0.1, cost = 80 → risk = 0.1 × 80 = 8   # Multiply the probability of failure by the cost of that failure to get a risk score.
Step 2: Money transfer: likelihood = 0.01, cost = 100 → risk = 0.01 × 100 = 1   # Even though the cost is maximum, the low likelihood reduces the risk score.
Step 3: Displaying account balance: likelihood = 0.05, cost = 30 → risk = 0.05 × 30 = 1.5   # The balance must be correct, but a failure is less harmful than losing money.
Step 4: New feature 'Add payee': likelihood = 0.5, cost = 60 → risk = 0.5 × 60 = 30   # High likelihood (new code is untested) and high cost drive the risk score up.
Step 5: Total risk = 8 + 1 + 1.5 + 30 = 40.5, but available effort is 20 hours.   # You cannot test everything to the same depth. You allocate effort proportionally to risk.
Step 6: Allocation: 'Add payee' gets 30/40.5 = 74% of effort = 14.8 hours; login gets 8/40.5 ≈ 20% = 4 hours; balance gets 1.5/40.5 ≈ 4% = 0.8 hours; money transfer gets 1/40.5 ≈ 2% = 0.4 hours   # Each feature gets a share of the time proportional to its risk score.

Result: You decide to spend 14.8 hours writing integration tests for the new 'Add payee' feature (since it is highest risk), 4 hours on login (mostly end-to-end testing, since it involves many components), and just 1.2 hours combined for the balance and money transfer features (they have low risk, so a few unit tests and a quick smoke test will do).


## From scratch, in Python

```python
# Let's compute the risk-based allocation from the example.
# We'll use a list of tuples. A tuple is like a list that cannot be changed.
# Each tuple holds (feature_name, likelihood, cost).

features = [
    ("login", 0.1, 80),
    ("money_transfer", 0.01, 100),
    ("balance", 0.05, 30),
    ("add_payee", 0.5, 60),
]

# We'll store results in another list of tuples (feature, risk_score)
risk_scores = []

# Loop over each feature tuple
for feature in features:
    name = feature[0]
    likelihood = feature[1]
    cost = feature[2]
    risk = likelihood * cost
    risk_scores.append((name, risk))

# Print the risk scores
print("Risk scores:")
for name, risk in risk_scores:
    print(f"{name}: {risk}")

# Sum all the risk scores
total_risk = 0
for name, risk in risk_scores:
    total_risk += risk

print(f"Total risk: {total_risk}")

# Available effort in hours
total_effort = 20

# Allocate effort proportionally to risk
print("\nEffort allocation (hours):")
for name, risk in risk_scores:
    fraction = risk / total_risk
    hours = fraction * total_effort
    print(f"{name}: {hours:.1f} hours")

```


## What the code does

The code starts by defining a list called `features`. Each item is a tuple, which is a container that holds several values in order. We have a tuple for each feature, with the name (a string), the likelihood (a number between 0 and 1), and the cost (a number). The line `for feature in features:` starts a loop that runs once for each tuple in the list. Inside the loop, we unpack the tuple into three variables: `name`, `likelihood`, and `cost` — so `name` becomes "login" on the first iteration, etc. We compute `risk = likelihood * cost` using plain arithmetic. Then we add a new tuple `(name, risk)` to the `risk_scores` list. The `append` method adds to the end of the list. After the loop, we print each risk score using a formatted string (`f"{name}: {risk}"`) which inserts the values into the text. Next, we sum the risk scores with another loop, adding each risk to `total_risk`. Finally, we compute the fraction and multiply by the total effort (20 hours) to get the hours for each feature. We use `:.1f` to show one decimal place. This is the exact plan we computed by hand — the code just automates it so we can change the inputs and instantly see the new allocation.


## Common misconceptions

**Test planning means writing a huge document nobody reads** — A test plan is just bureaucracy. I'll skip it and just start writing tests as I go.

Without a plan, you test whatever comes to mind first, which is usually the wrong thing. You might write dozens of unit tests for the trivial login helper and completely miss the new payment feature that is highly likely to break and costs the most if it does. Effort is wasted in the wrong places, and critical risks go unaddressed. In fact: The real test plan is a living checklist that says: 'These are the top 5 risks, and this is how much time each gets.' It can be a single page or a table. Its purpose is to force the risk-based decision, and to be reviewed and updated. A shared checklist is far more useful than an unread document.

**All tests should be equally deep** — Every function deserves the same amount of testing effort — I need 100% coverage on everything.

Equal coverage is impossible and wasteful. Some code is trivial (a simple getter) and some is critical (the password reset flow). Over-testing trivial code wastes hours that could catch a real bug in critical code. Under-testing critical code leaves you exposed. Coverage alone is not the goal — risk reduction is. In fact: Depth should always be proportional to risk. Critical, failure-prone features get heavy testing (multiple levels, many cases). Stable, low-risk features get a smoke test only. 80% of the effort should go to 20% of the features that carry 80% of the risk.

**"Level" only means unit vs end-to-end, nothing else** — The only testing levels I need are unit and end-to-end. Integration is just a type of end-to-end.

Integration testing is a distinct level that checks the interfaces between components — the contract. Without it, you either catch a bug too early (unit tests can't see mismatched interfaces) or too late (end-to-end tests are slow and hard to diagnose). Integration tests are fast and pinpoint the exact breakage, as your notes on Contract Testing say. In fact: Levels form a spectrum: unit (single function), integration (two or more modules together, often with mocks), and end-to-end (the whole system via a real UI or API call). Choose the level based on what you need to verify and how fast you need feedback. Prefer the least expensive level that catches the bug.

**A test plan is fixed once written** — I wrote the plan at the start of the sprint; now I just execute it. If a new feature is added, that's for the next sprint.

Software changes constantly — new features, bug fixes, changed requirements. If the plan is rigid, it becomes outdated and misses the new risks. For example, if a new feature is added mid-sprint that touches the payment module, you must re-evaluate its risk and adjust the plan, or you'll release an untested critical path. In fact: The plan is a living document. Any time requirements change or a defect is found, you re-run the risk assessment: 'Is this new feature high risk? Does this bug reveal other similar problems?' Adjust the allocation of effort accordingly. Agile development expects the plan to evolve with each iteration.

**Risk can only be assessed by a senior tester** — I'm a junior developer, I can't estimate likelihood and cost — I should just test everything equally.

Risk estimation is an informed guess, not an exact science. Even a junior can make a first guess based on the complexity of the code, whether it's new, and how many users touch it. The value is in making the guess explicit and discussing it as a team. Not trying to estimate at all is a certain way to waste effort. In fact: Everyone on the team can contribute to a risk list. Use your knowledge of the code and the product. Write down your estimate, and ask a senior to sanity-check. The exercise of writing down likelihood and cost forces you to think clearly, and the team can refine it together — the plan's precision comes from discussion, not individual genius.


## In short

Testing is not about checking everything; it is about deciding what matters most and spending your limited effort there. Use the formula risk = likelihood × cost to score each feature. Allocate effort proportionally to that risk. Choose the level (unit, integration, end-to-end) that gives you the fastest reliable feedback for the risk you are covering. A test plan is a living checklist, not a ceremony document. When a plan is missing, you end up testing the wrong things — or nothing — and the failures that hit users are exactly the ones you forgot.


## Where this is used

Test planning is used everywhere in AI systems. Here's specifically how:

**Large Language Models (LLMs):** A test plan for an LLM-based product (like a chat assistant) is enormous. You cannot test every possible prompt, so you plan a risk-based set of evaluation cases. High-risk prompts might be those that ask for medical or financial advice, or that trigger combative responses — you test these with a specific set of adversarial prompts. The plan also includes testing at the unit level (the model's response to a single prompt with a strict schema), integration level (the model's output feeding into a database or API), and end-to-end (the full user conversation through the UI). Risk drives depth: a model that will be used in healthcare gets extensive safety testing for hallucinations; a model for casual chat gets less.

**Small language models on devices (e.g., phone keyboards):** Here, testing effort is constrained by compute and battery. The plan might prioritise tests for latency (does the model respond within 50ms on an old phone?) and for privacy (does the local model ever leak data?). You test integration with the operating system, and you do end-to-end tests on a few physical devices — but you can't test every possible phone model. The plan selects a representative set of devices based on risk: most popular device model gets priority.

**Retrieval-Augmented Generation (RAG):** This system combines a retrieval component (e.g., semantic search over docs) with an LLM. Test planning here is critical because the failures are multi-level. You plan unit tests for the retriever's similarity ranking (does the right document come up for a query?), and integration tests for whether the LLM correctly uses the retrieved context. The plan must decide how many test documents and queries to use, and which retrieval failure modes (empty results, irrelevant results) are highest risk. You might use the pairwise testing approach from your library notes to choose parameter combinations (chunk size, embedding model, top-k) that are most likely to interact.

**Agentic systems (AI agents that plan and call tools):** These are the most complex to test. The plan must cover each tool call (unit test: does the agent parse the tool's response correctly?), the orchestration (integration: does the agent choose the right tool at each step?), and the whole trajectory (end-to-end: does the agent achieve the user's goal?). Risk prioritisation: a banking agent's high-risk action is a wrong transfer; a travel agent's high-risk action is booking the wrong flight. The plan allocates more effort to testing the decision branch with the highest cost of failure.

**Everyday applied ML (recommendation, vision, forecasting, fraud detection):** In a recommendation system, a test plan decides how to test the ranking algorithm (unit: does the score change when a feature changes?), the end-to-end user experience (does the recommendation appear and work?), and the offline evaluation (historical data). For a forecasting model, the risk is large errors — the plan allocates tests for different time horizons. Fraud detection tests must cover edge cases (unusual but valid transactions). The plan also decides the level: unit tests for the model code, integration tests for data pipelines, and end-to-end for the alert system.

**What a developer actually writes:** In practice, a developer creates a test plan as a document (often a shared checklist or a table in a project tracker) with columns: Feature, Level, Depth, Priority, Owner. They then write test code that implements these cases. For Python, they use `pytest` as the framework, writing functions like `def test_login_with_valid_credentials():`. The plan informs how many test functions, which are marked as `@pytest.mark.slow` and skipped in fast CI, and which are `@pytest.mark.e2e` and only run on release nights. If the plan is poorly done, the team either under-tests (critical bugs slip through) or over-tests (wasted effort that slows development, and the test suite becomes so large it is rarely run — which is worse than missing tests, as flaky tests train the team to ignore failures).


---

Read this on the web: https://basavapurushottam.com/learn-ai/test-planning
