# Pairwise Testing

Covering all pairs of parameter values rather than all combinations, on the evidence that most defects involve at most two factors.

**Rests on:** Equivalence Partitioning, Test Planning

*Software Engineering · Software Testing*


## Why learn this

This unlocks systematic test design: you'll be able to choose which test cases to write when the number of possible combinations is huge — and it is almost always huge. Pairwise testing is the skill that turns an impossible list of test cases into a short, practical one that still catches nearly all bugs. It sits beside equivalence partitioning and boundary value analysis in your library under Software Testing, and it feeds directly into Test-Driven Development and CI/CD: a pairwise-generated suite is what you put into a pipeline that must run fast. Concretely, after this concept you can look at a login form with five input fields and say 'I need about 25 cases, not 240', you can use a free tool to generate them, and you can explain to your team why that is enough.


## The intuition

Imagine you are testing a web form with three dropdowns, each having four options. Every possible combination is 4 × 4 × 4 = 64 cases. That is already a lot to type into a form manually. Add a fourth dropdown with four options, and you get 256 cases. Add a fifth, and it is 1024. Testing every combination is called 'exhaustive testing' — it explodes so quickly that nobody does it for real software.

But here is the key observation from decades of testing: most bugs show up when two specific values are used together, not when three or more specific values collide. A bug like 'if the country is India and the currency is USD, the page crashes' involves exactly two fields. A bug that only appears when India, USD, and a gold membership are all selected at once is vanishingly rare — and if it exists, you probably will not find it by chance anyway.

Pairwise testing says: make sure every possible pair of values appears in at least one test case. If you cover every pair, you cover the conditions for most bugs. You do not need to test every triple or every quadruple.

For the form with three dropdowns of four options each, there are 4 × 4 = 16 possible pairs between the first two dropdowns, and another 16 between each other combination of two dropdowns, making 48 pairs in total. But one test case covers three pairs at once: the case (Dropdown1=Red, Dropdown2=Small, Dropdown3=Cat) covers the pair (Red, Small), the pair (Red, Cat), and the pair (Small, Cat). So you can cover all 48 pairs with surprisingly few test cases — in fact, for this example, 16 test cases are enough. Pairwise testing is the art of choosing those minimal set of cases.


## An analogy

Think of a marriage bureau that matches people based on three attributes: religion (Hindu, Muslim, Christian), city (Delhi, Mumbai, Chennai), and profession (Engineer, Doctor, Teacher). To check that every type of match is workable, the bureau could interview all 3 × 3 × 3 = 27 possible combinations. But that is exhausting.

Instead, the bureau decides to check every pair of attributes: every combination of religion and city, every religion and profession, and every city and profession. There are 9 religion–city pairs, 9 religion–profession pairs, and 9 city–profession pairs, making 27 pairs in total. But each interviewed couple covers three pairs at once. For example, a couple with (Hindu, Delhi, Engineer) covers the pairs (Hindu, Delhi), (Hindu, Engineer), and (Delhi, Engineer). So the bureau only needs about 9 interviews to cover all pairs.

The analogy breaks down, though, because a real bureau might care about all three attributes together — a Hindu Engineer in Delhi might have special needs that a Hindu Engineer in Chennai does not. In software testing, that kind of triple interaction is rare. Most bugs come from a single setting or a pair of settings. The bureau stops being a good analogy exactly where the real world has genuine three-way dependencies; in testing, pairwise is a deliberate trade-off, accepting that we might miss a very specific triple bug because that bug is so unlikely. And unlike the bureau, the software test cannot just ask the couple — it has to run a test case, and running a test case has a real cost of time and money.


## Definition

Pairwise testing is a test design technique that selects a set of test cases such that every possible combination of values for any two parameters appears in at least one test case, dramatically reducing the number of cases compared to exhaustive testing while still catching the vast majority of interaction defects.


## How it connects

You have notes on Equivalence Partitioning and Boundary Value Analysis — those help you choose which values to test for a single parameter. Pairwise testing is the next step: after you have chosen the values for each parameter, pairwise testing tells you which combinations of those values to actually run. It fits neatly under your Software Testing concept, right alongside the test pyramid: pairwise testing is a way to design fast, focused tests that sit at the bottom of the pyramid. In your library, Contract Testing and API Testing are places where pairwise testing is especially useful, because an API endpoint often has many parameters (headers, query strings, body fields), and you need to test how they interact. Exploratory testing, by contrast, is the unplanned approach — pairwise testing is the planned, systematic alternative. Reference Testing and Property-Based Testing are different techniques that also try to cover input space, but with different goals: reference testing pins output, property-based testing generates random cases. Pairwise testing is complementary — you can use it to choose a fixed, manageable set of cases before you even start exploring.


## The mathematics

Pairwise testing relies on a simple branch of mathematics called combinatorics, which is just the study of counting. You already know how to count combinations: if you have three dropdown menus, and each menu has four options, the total number of full combinations is 4 times 4 times 4, which equals 64. Pairwise testing uses this same multiplication logic, but it asks a different question. Instead of asking 'how many full combinations are there?', it asks 'how many pairs of values exist?' and 'what is the smallest number of test cases needed to see every pair at least once?'

The key insight is that one test case covers multiple pairs at the same time. Imagine a test case where you set three parameters: Parameter A, Parameter B, and Parameter C. This single test case does not just check one combination; it checks three specific pairs simultaneously:
1. The pair formed by the value of A and the value of B.
2. The pair formed by the value of A and the value of C.
3. The pair formed by the value of B and the value of C.

Because each test case covers three pairs, you need far fewer test cases than the total number of full combinations. The goal is to choose a set of test cases so that every possible pair appears at least once. This problem is known as 'covering array design.' While finding the exact best set of test cases can be hard for computers, we can calculate a theoretical minimum using simple arithmetic.

Here is how we calculate that minimum:

First, let us define our variables in words so they are clear:
- Let m be the total number of parameters (for example, m = 3).
- For each parameter i (where i goes from 1 to m), let n_i be the number of possible values for that parameter.

Step 1: Count the total number of distinct pairs.
A pair is defined by two different parameters. If we pick Parameter i and Parameter j, the number of possible value-pairs between them is n_i multiplied by n_j. We must do this for every unique pair of parameters.

To write this clearly without complex symbols, we say:
The total number of pairs, which we will call P, is the sum of (n_i times n_j) for every unique combination of two different parameters i and j.

For example, if you have three parameters with sizes 4, 4, and 4:
- The pair of Parameter 1 and Parameter 2 has 4 times 4 = 16 possible value-pairs.
- The pair of Parameter 1 and Parameter 3 has 4 times 4 = 16 possible value-pairs.
- The pair of Parameter 2 and Parameter 3 has 4 times 4 = 16 possible value-pairs.
Total P = 16 + 16 + 16 = 48.

Step 2: Determine how many pairs one test case can cover.
If you have m parameters, a single test case sets a value for all of them. The number of unique pairs within those m values is the number of ways to choose 2 items from m. In mathematics, this is often written as 'm choose 2'. For our reader, we calculate it as:
(m times (m minus 1)) divided by 2.

For m = 3:
(3 times 2) divided by 2 = 6 divided by 2 = 3 pairs per test case.

Step 3: Calculate the lower bound.
The minimum number of test cases needed is at least the total number of pairs (P) divided by the number of pairs one test case can cover. Since you cannot have a fraction of a test case, we round up to the next whole number. This operation is called 'ceiling'.

Minimum test cases = ceiling(P divided by ((m times (m minus 1)) divided by 2)).

In our example:
48 divided by 3 equals 16.
So, you need at least 16 test cases. Tools like ACTS or Pairwise Online are designed to find a set of test cases that meets this minimum or comes very close to it.


## A worked example

We have three parameters: A with values A1, A2, A3, A4; B with values B1, B2, B3, B4; C with values C1, C2, C3, C4. We want a set of test cases (each is a triple (A, B, C)) such that every pair (A_i, B_j), every pair (A_i, C_k), and every pair (B_j, C_k) appears at least once. We will work out how many pairs there are, and then build a covering set.

Step 1: Number of (A,B) pairs = 4 × 4 = 16   # We count all combinations of one value from A and one from B.
Step 2: Number of (A,C) pairs = 4 × 4 = 16   # Same for A and C.
Step 3: Number of (B,C) pairs = 4 × 4 = 16   # And for B and C.
Step 4: Total number of distinct pairs = 16 + 16 + 16 = 48   # Each pair belongs to exactly one pair of parameters.
Step 5: Each test case covers 3 pairs: (A,B), (A,C), and (B,C)   # For example, the case (A1, B1, C1) covers (A1,B1), (A1,C1), and (B1,C1).
Step 6: Minimum possible number of test cases = ceil(48 / 3) = 16   # We need at least 16 cases because each covers at most 3 new pairs. This lower bound is achievable here.

Result: We need at least 16 test cases. The actual covering set can be constructed by a tool, and it will indeed use 16 cases. For this example, a valid 16-case set is: (A1,B1,C1), (A1,B2,C2), (A1,B3,C3), (A1,B4,C4), (A2,B1,C2), (A2,B2,C1), (A2,B3,C4), (A2,B4,C3), (A3,B1,C3), (A3,B2,C4), (A3,B3,C1), (A3,B4,C2), (A4,B1,C4), (A4,B2,C3), (A4,B3,C2), (A4,B4,C1). Check: for each A_i, all B_j appear exactly once, and all C_k appear exactly once, so every (A,B) and (A,C) pair is covered. Similarly for (B,C).


## From scratch, in Python

```python
import itertools

# Parameters and their possible values
params = {
    'A': ['A1', 'A2', 'A3', 'A4'],
    'B': ['B1', 'B2', 'B3', 'B4'],
    'C': ['C1', 'C2', 'C3', 'C4']
}

# 1. Generate all possible test cases (cartesian product)
all_cases = list(itertools.product(params['A'], params['B'], params['C']))
print('Total exhaustive cases:', len(all_cases))

# 2. Compute all possible pairs
pair_types = list(itertools.combinations(params.keys(), 2))
pairs_needed = set()
for p1, p2 in pair_types:
    for v1 in params[p1]:
        for v2 in params[p2]:
            pairs_needed.add((p1, v1, p2, v2))

print('Number of distinct pairs to cover:', len(pairs_needed))

# 3. Greedy algorithm to pick minimal covering set
def covers(test_case):
    '''Return a set of pairs that this test case covers.'''
    covered = set()
    for p1, p2 in pair_types:
        v1 = test_case[list(params.keys()).index(p1)]
        v2 = test_case[list(params.keys()).index(p2)]
        covered.add((p1, v1, p2, v2))
    return covered

selected = []
uncovered = set(pairs_needed)
while uncovered:
    # Pick the case that covers the most uncovered pairs
    best = None
    best_cover = set()
    for case in all_cases:
        cov = covers(case)
        newly = cov & uncovered
        if len(newly) > len(best_cover):
            best = case
            best_cover = newly
    selected.append(best)
    uncovered -= best_cover
    # Remove all cases that are now fully covered? Not necessary, but we can avoid re-picking same case
    all_cases.remove(best)

print('Selected test cases:')
for case in selected:
    print(case)
print('Number of pairwise cases:', len(selected))

# 4. Verify every pair is covered
covered_now = set()
for case in selected:
    covered_now |= covers(case)
print('All pairs covered?', covered_now == pairs_needed)
```


## The same thing, with a framework

```python
A popular library is `allpairspy`. You install it with `pip install allpairspy` and use `AllPairs`, which implements a more efficient algorithm. Here is the same example: from allpairspy import AllPairs; parameters = [['A1','A2','A3','A4'], ['B1','B2','B3','B4'], ['C1','C2','C3','C4']]; print(list(AllPairs(parameters))). The library does the same search but is heavily optimized for many parameters. Under the hood it uses a greedy algorithm like the one above, but with additional heuristics to handle larger problems.
```


## What the code does

The code starts by defining the parameters as a dictionary, where each key is a parameter name and each value is a list of allowed values. The line `itertools.product` generates every single combination — that is 4*4*4 = 64 cases. Then we use `itertools.combinations(params.keys(), 2)` to get all pairs of parameter names: (A,B), (A,C), (B,C). For each pair, we loop through all values and create a tuple of (parameter1, value1, parameter2, value2) to represent a pair to be covered. We store these in a set so duplicates are removed. Then we define a function `covers(test_case)` that, given a test case (a tuple of values), returns a set of all pairs it covers, using the same parameter names. The greedy loop repeatedly picks the test case that covers the most uncovered pairs, removes it from the list, and subtracts its covered pairs from the uncovered set. This continues until every pair is covered. Finally, we verify that the selected cases indeed cover every pair. The library `allpairspy` does exactly this but with a fancier algorithm called 'in-parameter-order' which is much faster for large numbers of parameters.


## Common misconceptions

**Pairwise testing means testing every pair by hand** — I have to think of every pair manually and write a test case for each.

That is tedious and error-prone for more than a few parameters. Tools like allpairspy or ACTS automatically generate a covering set, which is typically much smaller than the number of pairs. In fact: You use a tool to generate the minimal set of test cases that cover all pairs. Your job is to define the parameters and values, and then let the tool do the combinatorial search.

**Pairwise testing is the same as exhaustive testing** — If I cover all pairs, I have tested every possible combination.

Not at all. Pairwise testing covers only pairs. A specific combination of three values (e.g., A1, B1, C1) may not appear as a test case. Exhaustive testing covers all triples, quadruples, and so on, which is exponentially larger. In fact: Pairwise testing deliberately skips combinations of three or more parameters because those bugs are rare. It is a compromise, not a substitute for exhaustive testing when safety-critical.

**Pairwise testing guarantees finding all bugs** — If I use pairwise testing, I will catch every defect.

No technique can guarantee catching all bugs. Pairwise testing is based on the empirical observation that most bugs involve at most two parameters, but exceptions exist. Also, bugs can come from a single parameter (which pairwise covers) or from environmental factors not in the parameter list. In fact: Pairwise testing is a practical heuristic that provides a high degree of coverage for a fraction of the cost. It is best used alongside other techniques like boundary analysis, exploratory testing, and property-based testing.

**Pairwise testing is only for web forms** — I only need pairwise testing for user interfaces with dropdowns.

Pairwise testing applies wherever you have multiple parameters that interact: API endpoints, configuration settings, database queries, LLM prompts, device combinations, and more. In fact: Any situation with two or more variables and a large combination space can benefit from pairwise testing. It is a general test design technique, not specific to UI forms.

**The number of test cases is always the number of pairs** — If I have 48 pairs, I need 48 test cases.

That would be true only if each test case covered exactly one new pair, but a test case with three parameters covers three pairs. So you need far fewer cases. In fact: Each test case covers multiple pairs, so the required number of test cases is much smaller than the total number of pairs. The exact minimum is a combinatorial optimization problem, and tools find it.


## In short

Pairwise testing covers every pair of parameter values, not every combination. It exploits the observation that most bugs involve at most two factors. Use tools like allpairspy to generate the minimal set. Keep it in your testing toolkit alongside equivalence partitioning and boundary value analysis, and remember it is a practical compromise, not a guarantee.


## Where this is used

In large language models (LLMs), pairwise testing is used in evaluating and testing the model's behavior across different prompt templates, temperature settings, and context lengths. For example, an LLM application might have parameters: prompt template (3 variants), temperature (5 values), max tokens (3 values). Instead of testing all 3×5×3=45 combinations, a pairwise set of about 10 runs can cover every effect of any two parameters. In practice, a developer writes a script that uses `allpairspy` to generate the combinations, then runs the LLM on each and checks for anomalies. In small language models running on phones, pairwise testing is used to test the model across different device models, operating system versions, and precision settings (FP16, INT8, etc.). Since the number of device and OS combinations is huge, pairwise testing ensures every pair of (device, OS), (device, precision), (OS, precision) is tested without exploding the test matrix. The developer writes a test harness that takes a pairwise-generated list of configurations and runs the model on each. In retrieval-augmented generation (RAG), the parameters include chunk size, overlap, embedding model, and retrieval top-k. Pairwise testing helps choose a minimal set of experiments to see how each pair interacts, e.g., chunk size and top-k. A researcher or engineer can generate pairwise combinations of these parameters and run the RAG pipeline on each to measure answer quality. In agentic systems that plan and call tools, the agent might have parameters: planner type, tool availability, memory length, and API rate limit. Pairwise testing is used to test the agent across different combinations to find interactions where a certain pair of settings crashes the agent or causes it to loop. A developer writes integration tests that instantiate the agent with each pairwise combination and asserts it completes a task without errors. In everyday applied machine learning, pairwise testing is used in feature engineering: when choosing which feature transformations to apply, you might have a pipeline with parameters like scaling method (min-max, z-score, none), encoding method (one-hot, label, none), and model type (linear, tree, neural). Instead of trying all combinations, pairwise testing selects a small set of pipelines to evaluate quickly, giving a good sense of which features matter without training dozens of models. In software development, pairwise testing is applied when testing API endpoints with many query parameters, such as sort order, limit, and filter. The developer generates pairwise cases and feeds them into an API test runner, catching bugs where two parameters interact badly, like 'limit=0 and sort=desc crashes the server'. The techniques from Full Stack Testing, 2nd edition, emphasize that testing is a full-stack skill, and pairwise testing is a core technique to have in your toolbox, applying to functional, performance, and security testing alike. The note about exploratory testing being organic — you should use pairwise testing to script the planned cases, and leave room for exploratory discovery. When done badly, pairwise testing can miss a bug that requires three parameters, but that risk is accepted because the cost of exhaustive testing is prohibitive. The key is to document the choice and be aware of the assumption.


---

Read this on the web: https://basavapurushottam.com/learn-ai/pairwise-testing
