# Bradley-Terry Model

A statistical model turning pairwise comparisons into latent scalar scores, the standard bridge from preferences to a reward function.

**Rests on:** Logistic Regression, Preference Data Collection

**Leads to:** Reward Modeling

*LLMs and Generative AI · Post-Training Alignment*


## Why learn this

This concept explains how computers learn what you like by watching you choose between two options. Every time you rate one answer as better than another, or click on one product instead of another, you are providing data that helps an AI understand your taste. This model turns those simple 'this is better than that' choices into a structured ranking system. Understanding it reveals how vague human feelings are converted into precise numbers that guide an AI's behavior. Without this step, an AI would not know how to weigh helpfulness against accuracy or which of two valid answers you prefer.

For example, consider a simple case: you prefer Answer A to Answer B, and Answer B to Answer C. The model must assign a single 'strength' number to each answer such that A is stronger than B, and B is stronger than C, while respecting the probabilistic nature of human judgment (i.e., sometimes B might win against A by chance). This model provides the specific numbers that allow an AI to learn which responses are 'good'.


## The intuition

Think of a chess rating system. When two players face each other, a stronger player is more likely to win, but not guaranteed — an upset happens, and the bigger the rating gap, the more lopsided the odds. The Bradley-Terry model does the same thing, but for any pair of options, not just chess players.

Give each option a single number (its 'strength' or 'score'), and the model says: the probability that Option A beats Option B depends only on the difference between their scores. 

Let's work through a concrete case with specific numbers. Imagine three items: Item 1, Item 2, and Item 3. We assign them hypothetical strengths of 10, 5, and 0 respectively.

The model calculates the probability that Item 1 (strength 10) beats Item 2 (strength 5) using a specific formula based on their difference. The higher the strength of A relative to B, the closer the probability gets to 1 (certainty). If Item A has strength 5 and Item B has strength 5, the probability is exactly 0.5 — a coin flip. If Item A has strength 2 and Item B has strength 8, the probability is very low.

The magic is that we don't know these scores initially — we only see the outcomes of pairwise comparisons (e.g., 'A beat B', 'C lost to D'). The model's job is to find scores that make the observed outcomes maximally likely. It does this by a simple iterative loop:
1. Start with guesses for the strengths (e.g., all zeros).
2. Compare each pair's predicted probability to what actually happened.
3. Nudge the scores up or down based on whether the prediction was too high or too low.
4. Repeat.

After enough iterations, the scores converge to values that best explain the preferences. This is the highway from 'people preferred this answer' to 'here is a number representing how good that answer is' — and that number is what a reward model uses to train an AI.


## An analogy

Imagine you are a cricket match referee who has never seen any team play, but you have a list of match results: India beat Australia, Australia beat England, England beat India (a circular mess). You know nothing about the teams' actual strengths, but you want to assign each team a single 'strength number' so that the results make sense.

Start with all strengths equal. Look at the first result: India beat Australia. With equal strengths, your model predicted a 50/50 chance. Since India actually won (a 'surprise' in your model's view), you nudge India's strength up a little and Australia's down a little.

Next result: Australia beat England. Now Australia's strength is up, so the model predicts Australia wins more often than 50/50, but not 100% — England still has a chance. When Australia does win, the model is less surprised, so the nudge is smaller.

Continue through all results, looping many times. Each nudge moves the scores in the direction that would have made the observed outcome more expected. After enough passes, the strengths settle: India and Australia might be close, England a bit lower. Now you can predict a match between any two teams — and the circular result (England beat India) turns out to be just a lucky upset, not a contradiction.

Where the analogy breaks: unlike cricket where a team's strength is roughly consistent, human preferences are inconsistent — people sometimes prefer A over B, B over C, and C over A. The Bradley-Terry model cannot represent that circularity; it will force a single ranking, which may not match every individual preference.


## Definition

The Bradley-Terry model is a statistical method that estimates a single numerical 'strength' for each item from pairwise comparison data, such that the probability one item is preferred over another depends only on the difference in their strengths, found by adjusting the strengths to maximize the likelihood of the observed preferences.


## How it connects

This concept sits alongside Reward Modeling and Direct Preference Optimization. Reward Modeling is the process of building a numerical scorecard for AI responses based on human preferences, which relies directly on the Bradley-Terry calculations described here. Direct Preference Optimization is a technique that trains an AI to follow these preferences by adjusting its outputs to align with the strength scores derived from the Bradley-Terry model, effectively bypassing the need for a separate scoring step. Both topics depend on the ability to translate relative choices into absolute strength values.


## The mathematics

The mathematical branch here is **probability** — the study of chance and how to measure uncertainty. You have met probability implicitly: a coin has a 50% chance of heads, rain has a 70% chance tomorrow. The Bradley-Terry model uses probability to say 'how likely is it that A beats B?' given their strengths. The core operation is the **sigmoid function** (also called the logistic function), which takes any number — negative, positive, huge — and squashes it into a number between 0 and 1, so it can be treated as a probability. The sigmoid is not new math; it is just a formula that does this squashing. To understand it, imagine you have a raw score: the difference between A's strength and B's strength. If A is much stronger, the difference is large and positive; the sigmoid turns that into a number close to 1 (A almost always wins). If B is stronger, the difference is negative; the sigmoid outputs near 0. If they are equal, the difference is 0, and the sigmoid outputs exactly 0.5 — a coin flip. The model then works backwards: given many observed outcomes, it tries different strength values, computes how likely the observed outcomes were under those strengths, and adjusts the strengths to make that likelihood as high as possible. This adjustment is done by a simple iterative process: for each comparison, multiply the probability of the observed winner by a small nudge, increase the winner's strength and decrease the loser's, then repeat. You do not need to know calculus — the algorithm is just repeated arithmetic. The only new idea is **iterative improvement**: start with guesses, make small changes that move in the direction of better fit, and repeat until the changes become tiny.

For two items i and j with strengths θᵢ and θⱼ:

P(i beats j) = 1 / (1 + e^(−(θᵢ − θⱼ)))

where e is the base of the natural logarithm (about 2.718), and the difference θᵢ − θⱼ is fed through the sigmoid function to give a probability between 0 and 1.

Learning rule (one iteration for a single comparison):
θᵢ ← θᵢ + α × (1 − P(i beats j))  (if i actually won)
θⱼ ← θⱼ − α × (1 − P(i beats j))  (if i actually won)

where α is the learning rate (a small positive number like 0.1). The same rule applies with the loser taking the negative nudge.


## A worked example

```
Three items: A, B, C. We have observed pairwise comparisons: A beats B, B beats C, and A beats C. Start with all strengths equal at 0.0. We will use a learning rate (call it alpha) of 0.1 — this controls how big each nudge is. The sigmoid function is: sigmoid(difference) = 1 / (1 + e^(−difference)), where e ≈ 2.718. We will work through one iteration of the algorithm to see how the strengths update.

Step 1: Initial strengths: sA = 0.0, sB = 0.0, sC = 0.0   # We start with equal strengths; the model has no information yet.
Step 2: Compare A vs B: difference = sA − sB = 0.0 − 0.0 = 0.0   # The difference in strengths is zero.
Step 3: Predicted probability A beats B: sigmoid(0.0) = 1 / (1 + e^−0.0) = 1 / (1 + 1) = 0.5   # Equal strengths give a 50% chance, as expected.
Step 4: Observed outcome: A beat B. Nudge: 0.1 × (1 − 0.5) = 0.05 → sA = 0.0 + 0.05 = 0.05, sB = 0.0 − 0.05 = −0.05   # Since A won, we increase A's strength and decrease B's. The magnitude of the nudge is the learning rate times the 'surprise' (how much the prediction differed from the outcome).
Step 5: Compare B vs C: difference = sB − sC = −0.05 − 0.0 = −0.05   # B is now slightly lower than C.
Step 6: Predicted probability B beats C: sigmoid(−0.05) = 1 / (1 + e^(0.05)) ≈ 1 / (1 + 1.0513) ≈ 0.4875   # B is slightly weaker, so its win probability is just under 0.5.
Step 7: Observed: B beat C. Nudge: 0.1 × (1 − 0.4875) = 0.05125 → sB = −0.05 + 0.05125 = 0.00125, sC = 0.0 − 0.05125 = −0.05125   # B won despite being predicted to lose, so we push B up and C down.
Step 8: Compare A vs C: difference = sA − sC = 0.05 − (−0.05125) = 0.10125   # A is now above C.
Step 9: Predicted probability A beats C: sigmoid(0.10125) ≈ 0.5253   # A is slightly stronger, so a bit above 0.5.
Step 10: Observed: A beat C. Nudge: 0.1 × (1 − 0.5253) = 0.04747 → sA = 0.05 + 0.04747 = 0.09747, sC = −0.05125 − 0.04747 = −0.09872   # A wins again, so A's strength rises further, C drops.

Result: After one iteration, the strengths are sA ≈ 0.097, sB ≈ 0.001, sC ≈ −0.099. A is ranked highest, C lowest, which matches the observed outcomes. Repeating many times would make the strengths converge to values that make the observed results maximally likely.
```


## From scratch, in Python

```python
import math

# Data: list of pairs (winner, loser) from observed comparisons
comparisons = [
    ('A', 'B'),
    ('B', 'C'),
    ('A', 'C'),
]

# Items and initial strengths
items = ['A', 'B', 'C']
strengths = {item: 0.0 for item in items}

# Learning rate
alpha = 0.1

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

# Run 10 iterations
for iteration in range(10):
    print('Iteration', iteration + 1, strengths)
    for winner, loser in comparisons:
        diff = strengths[winner] - strengths[loser]
        prob = sigmoid(diff)  # probability winner beats loser according to current strengths
        # Update: increase winner, decrease loser
        step = alpha * (1 - prob)
        strengths[winner] += step
        strengths[loser] -= step

print('Final strengths:', strengths)
```


## The same thing, with a framework

```python
import numpy as np

# Using NumPy: strengths as a vector, comparisons as index pairs
items = ['A', 'B', 'C']
strengths = np.zeros(3)  # array of zeros
# index map: A->0, B->1, C->2
idx = {item: i for i, item in enumerate(items)}

comparisons = [(0, 1), (1, 2), (0, 2)]  # pairwise indices

alpha = 0.1
for _ in range(10):
    for i, j in comparisons:
        diff = strengths[i] - strengths[j]
        prob = 1 / (1 + np.exp(-diff))
        step = alpha * (1 - prob)
        strengths[i] += step
        strengths[j] -= step

print(strengths)
```


## What the code does

The code begins by setting up a list of comparisons, each a pair (winner, loser). A dictionary is used to store strengths, with item names as keys and numbers as values — a dictionary is like a phonebook: look up a name to get a number. The sigmoid function is defined: it takes any number and returns a value between 0 and 1. The main loop runs 10 iterations. In each iteration, it processes every comparison: it computes the difference in strengths, feeds that through sigmoid to get a predicted win probability, then updates the strengths. The update uses the learning rate alpha multiplied by (1 - prob) — this is the 'surprise': if prob is close to 1 (we were confident the winner would win), the step is small; if prob is close to 0 (upset), the step is large. The winner gets a plus, the loser a minus. After many iterations, the strengths converge. The NumPy version does the same thing but uses a vector (a list-like structure designed for fast arithmetic) and index numbers instead of item names — under the hood it is the identical multiply-and-add loop.


## Common misconceptions

**The Bradley-Terry model is only for chess ratings** — I've seen it used for sports and games, so it can't apply to language models.

The model is agnostic to what the 'items' are. It only needs pairwise comparisons, which are exactly what preference annotators produce when choosing between two model responses. In fact: The Bradley-Terry model is a general method for turning pairwise preferences into scores, used in LLM reward modeling, search ranking, recommender systems, and more.

**The scores are directly the probability of winning** — If item A has a score of 3 and item B has a score of 2, then A beats B 60% of the time.

The probability depends on the *difference* of the scores through the sigmoid function, not the raw ratio. A difference of 1 gives about 73% chance, not 60%. In fact: The model uses the difference in scores, passed through the sigmoid function, to get a probability. The scores themselves are on an arbitrary scale.

**The model can handle circular preferences** — My data has A over B, B over C, C over A — the model will figure out that all three are good.

The Bradley-Terry model assumes transitivity: if A beats B and B beats C, then A must be stronger than C. It cannot represent a cycle; it will force a total order that contradicts some of the data. In fact: The model assumes a single scalar strength per item, which imposes a transitive ranking. Human preferences often violate this, so the model will not perfectly fit circular data — it will find the best compromise.

**The learning rate determines the final scores** — If I use a bigger learning rate, I get more accurate scores.

A learning rate that is too high can cause the algorithm to overshoot and oscillate or diverge, never settling on a good ranking. The final converged scores should be similar across learning rates if the learning rate is small enough; the learning rate only affects the path and speed of convergence. In fact: The learning rate controls step size. A small rate leads to slow but stable convergence; too large a rate can cause the scores to bounce around or blow up. With many iterations, a small rate finds a good solution.

**The model requires a fixed number of comparisons per item** — I need every item to be compared against every other item equally, otherwise it won't work.

The model works with any set of pairwise comparisons, even if some items appear many times and others few. It simply adjusts strengths to fit the data it has, and items with fewer comparisons have more uncertainty in their estimated strength. In fact: You can have uneven comparison counts. The model estimates strengths from whatever pairs are observed; more data leads to more precise estimates.

**It is a machine learning model like a neural network** — It's just another kind of neural net that I train with backpropagation.

The Bradley-Terry model is a simple statistical model with one parameter per item. It is not a neural network. However, it can be *embedded* in a neural network training pipeline: the output of a neural net is plugged into the Bradley-Terry probability, and then the whole thing is trained with gradient descent. In fact: The Bradley-Terry model is a statistical model. When used in reward modeling, the 'strength' is the output of a neural network, and the Bradley-Terry probability forms the loss — but the model itself is just the sigmoid of a difference.


## In short

The Bradley-Terry model estimates a single score per item from pairwise comparisons. It assumes the probability that i beats j is sigmoid(sᵢ − sⱼ), where sigmoid squashes the difference into a 0–1 probability. The algorithm iteratively nudges strengths in the direction that makes observed winners more likely, using a learning rate. It is the mathematical foundation of reward modeling in LLM alignment, and its assumptions (transitivity, scalar strength) must be kept in mind when applying it to human preferences.


## Where this is used

In **large language models**, the Bradley-Terry model is the backbone of reward modeling. During post-training, human annotators are shown two responses to the same prompt and pick the better one. These pairwise comparisons are fed into a model (often a smaller transformer) that learns to score any response with a single number. The training objective for that reward model is exactly the Bradley-Terry likelihood: maximize the probability that the chosen response scores higher than the rejected one. The reward model's scalar output is then used in RLHF (reinforcement learning from human feedback) to fine-tune the language model — the model is optimized to produce responses that get high reward scores, which proxies for human approval. Without the Bradley-Terry model, there would be no principled way to turn discrete human clicks into a continuous reward signal. In **small language models** running on phones or laptops, the same preference optimization happens, but the reward model is often distilled or the preference learning is done directly (as in DPO). The Bradley-Terry assumption still underlies the math — DPO derives its loss from the Bradley-Terry reward model objective, just without explicitly training a separate reward model. In **retrieval-augmented generation**, the Bradley-Terry model appears in re-ranking: after retrieving candidate documents, a model that has been trained on pairwise preferences (which document is more relevant) scores each one, and the top-rated documents are fed to the generator. The re-ranker was trained on human or model judgments of pairwise relevance, again using Bradley-Terry. In **agentic systems** that plan and call tools, the model may be trained with preference optimization on trajectories — the agent tries different action sequences, and a human or a judge model says which sequence was better. Each pairwise comparison updates the policy using the same Bradley-Terry probability, either through a reward model or directly in the loss. In **everyday applied machine learning**, the Bradley-Terry model appears in recommender systems (implicit feedback: which item did the user click or buy), in search ranking (which result was more relevant), in sports analytics, and in any 'A/B test' analysis where you compare two options and need to estimate which is better under noise. A developer using this today would reach for libraries like `pytorch` or `tensorflow` for neural reward models, or `numpy`/`scipy` for a simple standalone Bradley-Terry implementation (e.g., `choix` or `pyltr` for learning-to-rank). They would write code that loads preference data as pairs, defines a model (either a fixed linear form or a neural net), and trains it with the Bradley-Terry loss. What breaks when it is done badly: if the preference data has many ties (annotators can't decide), the model becomes uncertain; if the data has systematic biases (annotators prefer longer responses), the reward model learns that bias and the language model gets reward hacked — it exploits the proxy rather than genuine quality. The developer must also watch for the transitivity assumption: the Bradley-Terry model assumes if A beats B and B beats C, then A should beat C, but human preferences often violate this — if the model is applied too rigidly, it will force a ranking that contradicts some human judgements.


---

Read this on the web: https://basavapurushottam.com/learn-ai/bradley-terry-model
