# Preference Data Collection

Gathering human comparisons between model outputs — the annotation process that determines everything downstream in alignment.

**Rests on:** Synthetic Data Generation

**Leads to:** Alignment Case Study, Bradley-Terry Model, ORPO and Preference Optimization, Reward Modeling

*LLMs and Generative AI · Post-Training Alignment*


## Why learn this

You are building an AI that needs to be reliable, not just fluent. Early training teaches the model to predict words; it does not teach it to follow instructions or avoid harm. To fix this, you need human input to show the model what 'good' looks like. If the human judgments you gather are poor, no amount of computing power will help the model become useful. The bottleneck in creating high-quality AI is not hardware, but the quality of these human comparisons. By the end of this page, you will know how to collect these judgments effectively.


## The intuition

When a model is first trained, it simply predicts the next word. This makes the text flow well, but it does not make the model helpful, honest, or safe. To achieve those qualities, we must show the model what 'good' looks like. We cannot write a precise rule for being helpful because the concept is too vague. Instead, we present the model with many pairs of answers — one correct and one incorrect — allowing it to learn the pattern. This act of gathering those pairs is called preference data collection. It serves as the raw material for alignment. Without it, the model has no definition of 'better.' With it, you can steer the model's behavior in almost any desired direction. The key insight is that these preferences are comparative, not absolute. We do not ask 'is this answer good?' but rather 'is this answer better than that one?' This comparative question is much easier for humans to answer reliably.


## An analogy

Imagine you are a chef teaching an apprentice how to cook. You cannot write a complete recipe for every possible dish, but you can taste two plates side by side and say 'this one is better.' Over many tastings, the apprentice learns your preferences — what you like and dislike. If you consistently prefer the professional chef's dishes over the apprentice's, the apprentice learns to aim for that higher standard. This mirrors how preference collection works: the model is the apprentice, the human annotator is the chef, and each comparison is one data point. A critical detail emerges when the apprentice only tastes their own food; they become more confident but do not improve. Similarly, if a model only compares its own previous outputs, it cannot learn beyond its existing style. This is why collecting data from the model's current generation process is essential — you must evaluate the apprentice's latest dishes regularly, not just the chef's older ones. The analogy has limits, as a chef adjusts in real time while a model learns from thousands of static comparisons, but the core mechanic of learning from pairwise judgments holds true.


## Definition

Preference data collection is the process of gathering human rankings between different model responses to the same prompt, where humans mark one response as preferred over another, creating the training signal that shapes the model's behavior.


## How it connects

This concept forms the foundation of the post-training stack. You are likely familiar with Supervised Fine-Tuning, which teaches a model to imitate good examples by showing it correct answers. Preference collection builds on this by teaching the model to distinguish better responses from worse ones on its own. The paired data you collect here is consumed by methods such as Direct Preference Optimization and Group Relative Policy Optimization, which adjust the model based on these comparisons. It also feeds into Reward Modeling, a technique that converts these pairwise judgments into numerical scores using the Bradley-Terry model, a statistical method for predicting the probability that one item is preferred over another. The distinction between training on live data generated by the current model versus fixed datasets is crucial here, as is the risk of reward hacking, a situation where the model learns to exploit flaws in the preference data rather than actually improving its helpfulness.


## The mathematics

Preference data collection is not just about people clicking buttons; it is a mathematical process of turning noisy human opinions into reliable numbers. We use probability and statistics because human judgments are inconsistent. One person might prefer response A today and response B tomorrow. To handle this, we treat the 'quality' of a response as a hidden number, called a latent score. We cannot see this score directly, but we can infer it by looking at which responses win more often.

The standard tool for this is called the Bradley-Terry model. It connects two things: the hidden quality scores and the probability that one response beats another. The rule is simple: the higher the score, the more likely the response is to be chosen. Specifically, if we have two responses, A and B, with scores s_A and s_B, the probability that A is preferred over B is calculated by taking the exponential of s_A (which means e raised to the power of s_A) and dividing it by the sum of the exponentials of both scores.

Let us write this out clearly. Let p(A beats B) be the probability that a random person prefers A over B. The formula is:
p(A beats B) = (e raised to the power of s_A) divided by ((e raised to the power of s_A) + (e raised to the power of s_B)).

Here, 'e' is a specific number, approximately 2.718. This formula ensures that if s_A is larger than s_B, the numerator is larger relative to the denominator, making the probability greater than 0.5 (or 50%). If s_A equals s_B, the probability is exactly 0.5.

To find these scores from real data, we use a method called maximum likelihood estimation. This sounds complex, but it just means we adjust the scores until the model's predicted probabilities match the actual observed wins as closely as possible. In practice, this is done by minimizing a 'loss' value. For each comparison where A won against B, the loss is calculated using the natural logarithm of the sigmoid function applied to the difference in their scores.

The sigmoid function, often written as sigma, takes any number and squashes it between 0 and 1. It is defined as:
sigma(x) = 1 divided by (1 + e raised to the power of negative x).

So, if A won against B, we look at the difference in their scores: s_A minus s_B. We plug this difference into the sigmoid function to get a predicted probability that A wins. The loss for this single comparison is the negative natural logarithm of that probability.

Why do we do this? Because minimizing the loss is mathematically equivalent to maximizing the likelihood that our model generated the observed data. If the model predicts A should win with 90% probability and A does win, the loss is small. If the model predicts A should lose but A wins, the loss is large, and the scores are adjusted.

Symbol glossary:
p(A beats B) — read 'p of A beats B', means the probability that a random annotator chooses A over B.
s_A — the latent quality score for response A. This is a hidden number we are trying to find.
s_B — the latent quality score for response B.
e — Euler's number, approximately 2.718. It is the base of natural logarithms.
log — the natural logarithm function, which is the inverse of raising e to a power.
sigma — the sigmoid function, which maps any real number to a value between 0 and 1.
y_w — the 'winner' response in a comparison pair.
y_l — the 'loser' response in a comparison pair.


## A worked example

Let us work through a concrete example. Suppose we have one prompt: 'Explain gravity to a child'. We have three candidate responses: A, B, and C. We ask 10 different people (annotators) to judge pairs of these responses. Each person sees two responses side-by-side and picks the better one.

Here are the results of the pairwise comparisons:
- In 10 comparisons between A and B, A was chosen 8 times.
- In 10 comparisons between A and C, A was chosen 6 times.
- In 10 comparisons between B and C, B was chosen 7 times.

We want to estimate the latent quality scores s_A, s_B, and s_C. While the full Bradley-Terry calculation requires solving a system of equations (which is done by computers), we can illustrate the logic using simple win rates and then show how the probability formula works for one pair.

Step 1: Calculate the raw win rate for A against B.
A won 8 times out of 10. So the observed probability that A beats B is 8 divided by 10, which equals 0.8.

Step 2: Calculate the raw win rate for A against C.
A won 6 times out of 10. So the observed probability that A beats C is 6 divided by 10, which equals 0.6.

Step 3: Calculate the raw win rate for B against C.
B won 7 times out of 10. So the observed probability that B beats C is 7 divided by 10, which equals 0.7.

Step 4: Infer relative scores from these probabilities.
The Bradley-Terry model tells us that the difference in scores corresponds to the log-odds of winning. Let's look at the comparison between A and B.
We observed that A beats B with probability 0.8.
The formula says: p(A beats B) = e^(s_A) / (e^(s_A) + e^(s_B)).
This can be rearranged to show that the ratio of the exponentials is related to the odds:
e^(s_A) / e^(s_B) = p(A beats B) / (1 - p(A beats B)).
For A vs B: The odds are 0.8 divided by (1 - 0.8), which is 0.8 / 0.2 = 4.
So, e^(s_A - s_B) = 4.
Taking the natural logarithm of both sides: s_A - s_B = log(4).
Using a calculator, log(4) is approximately 1.386.
This means A's score is about 1.386 points higher than B's score.

Step 5: Do the same for B vs C.
We observed that B beats C with probability 0.7.
The odds are 0.7 divided by (1 - 0.7), which is 0.7 / 0.3 = 2.333...
So, e^(s_B - s_C) = 2.333.
Taking the natural logarithm: s_B - s_C = log(2.333).
Using a calculator, log(2.333) is approximately 0.847.
This means B's score is about 0.847 points higher than C's score.

Step 6: Combine the differences to find relative scores.
Let us set s_C to 0 as a reference point (since only differences matter).
Then s_B = 0 + 0.847 = 0.847.
And s_A = s_B + 1.386 = 0.847 + 1.386 = 2.233.

So our estimated latent scores are:
s_A ≈ 2.23
s_B ≈ 0.85
s_C = 0.00

Step 7: Verify the model with these scores.
Let's check if these scores predict the original win rate for A vs B (which was 0.8).
Using the formula: p(A beats B) = e^(2.233) / (e^(2.233) + e^(0.847)).
e^(2.233) is approximately 9.33.
e^(0.847) is approximately 2.33.
The denominator is 9.33 + 2.33 = 11.66.
The probability is 9.33 / 11.66 ≈ 0.80.
This matches our observed win rate of 0.8 exactly (within rounding error).

Result: The latent scores confirm that A is the strongest response, followed by B, then C. The gap between A and B (1.39) is larger than the gap between B and C (0.85), which explains why A's win rate against B (80%) was higher than B's win rate against C (70%). In a real system, these calculations are performed simultaneously for all pairs using optimization algorithms to find the set of scores that minimizes the total error across all comparisons.


## From scratch, in Python

```python
import random

# We have responses for one prompt. Each response is a string.
responses = {
    'A': 'Explain gravity as a force that pulls things together.',
    'B': 'Gravity is a fundamental force that attracts bodies with mass.',
    'C': 'In Newtonian physics, gravity is an inverse-square force between masses.'
}

# We'll simulate 10 annotators, each seeing a pair and choosing one.
# Real collection uses actual humans; this code mimics it.
comparisons = []
pairs = [('A','B'), ('A','C'), ('B','C')]

# Simulate annotator preferences according to a known quality order: A > B > C
# We'll use a simple probabilistic model: each annotator picks the 'better' with 80% chance, else the other.
random.seed(42)
for _ in range(10):
    for x, y in pairs:
        # decide true winner based on quality order
        true_winner = x if (x < y) else y  # alphabetically lower is assumed better for simplicity
        # 80% chance to pick true winner
        if random.random() < 0.8:
            chosen = true_winner
            rejected = y if x == true_winner else x
        else:
            chosen = y if x == true_winner else x
            rejected = true_winner
        comparisons.append((chosen, rejected))

# Count wins for each response
wins = {'A':0, 'B':0, 'C':0}
total = {'A':0, 'B':0, 'C':0}
for chosen, rejected in comparisons:
    wins[chosen] += 1
    total[chosen] += 1
    total[rejected] += 1

# Compute win rates
win_rates = {resp: wins[resp]/total[resp] for resp in responses}
print('Win rates:', win_rates)
# Output (approx): A 0.733, B 0.55, C 0.417 (depends on random seed)

# The dataset for training is a list of tuples (prompt, chosen, rejected)
# In practice, you'd record the prompt along with the pair.
preference_dataset = []
for chosen, rejected in comparisons:
    # We don't have a stored prompt; use a placeholder
    preference_dataset.append(('Explain gravity to a child', responses[chosen], responses[rejected]))
print('Total comparisons:', len(preference_dataset))
print('First three examples:')
for ex in preference_dataset[:3]:
    print(ex)

# In a real system, you'd store the raw texts and labels.
# That's the core of preference data collection.
```


## The same thing, with a framework

In practice, you'd use a library like Hugging Face's `datasets` to store and load the data, and PyTorch to train a reward model. Here's a sketch:

```python
from datasets import Dataset, Features, Value, Sequence

# Suppose you have a list of (prompt, chosen, rejected) tuples
data = [
    ('Explain gravity to a child', 'Gravity pulls things together.', 'Gravity is a force of attraction between objects with mass.'),
    # ... more examples
]

# Create a Hugging Face Dataset object
features = Features({
    'prompt': Value('string'),
    'chosen': Value('string'),
    'rejected': Value('string'),
})
dataset = Dataset.from_list(data, features=features)

# This dataset can be used to train a reward model with PyTorch.
# Under the hood, the reward model computes scores for chosen and rejected, 
# and the loss is the negative log-sigmoid of the difference, as in the formula.
```

What the framework does underneath: the same counting and comparison logic, but with neural networks. Each response is encoded into a vector, scores are computed from those vectors, and the loss pushes the chosen score above the rejected score.


## What the code does

The from-scratch code simulates the collection process. First, we define three responses. Then we loop 10 times over each pair, randomly picking a winner with 80% chance following a hidden quality order. We store each chosen–rejected pair. We then count wins and total appearances for each response, compute win rates, and build a dataset of (prompt, chosen, rejected) tuples. The framework version shows how you'd store the real data in a Hugging Face `Dataset`, and how the reward model loss uses the scores. The key is that the data structure — list of tuples — is the foundation for everything downstream.


## Common misconceptions

**Preference data is just gathering 'good' examples.** — I just need a bunch of good responses and a bunch of bad responses and I'm done.

Preference is inherently pairwise: an annotator sees two responses to the same prompt and picks one. A collection of individually good examples doesn't give the model a signal about relative quality — it's the contrast that teaches. Without paired comparison, the model learns to imitate, not to rank. In fact: Preference data is always in the form of paired comparisons — for each prompt, at least one chosen and one rejected response. The chosen response is preferred over the rejected one, and that relative judgement is the training signal.

**More annotators always mean better data.** — If I hire a thousand people to label, the data will automatically be high-quality.

Annotator disagreement, inconsistent guidelines, and fatigue degrade data quality. Without clear instructions and quality checks, a large crowd produces noisy, contradictory labels that confuse the reward model. More data doesn't fix bad labelling. In fact: Quality depends on consistent guidelines, careful annotator training, and measuring inter-annotator agreement. You need enough annotators to average out noise, but only if the instructions are unambiguous.

**On-policy vs off-policy doesn't matter much.** — Any preference data is fine, I can reuse old data from a different model.

If the preference data comes from outputs of an old policy (off-policy), the current model's outputs look very different, so the learned preferences don't transfer well. This is the on-policy vs off-policy problem: training on off-policy data causes distribution drift. In fact: Preference data should be collected from the current policy's own outputs whenever possible. That's why modern pipelines sample from the current model, get human judgements on those samples, then update — repeating the cycle.

**Preference data is only for RLHF.** — Preference data is only useful for reinforcement learning, not for other methods.

Preference data is used in DPO, ORPO, and even as direct training data for reward models that supervise later stages. It's the universal currency of alignment — any post-training that wants to optimise against human preferences needs it. In fact: Preference data feeds reward modeling, DPO, ORPO, GRPO, and even pure supervised fine-tuning on chosen responses. It's not RL-specific; it's the foundation of all preference-based alignment.


## In short

Preference data collection is the human-driven process of gathering pairwise comparisons between model outputs. Its quality determines alignment quality. Collect on-policy, write clear guidelines, check annotator agreement, and track fatigue. This data is the seed for every downstream alignment algorithm — reward models, DPO, GRPO — and the single most important thing you can get right.


## Where this is used

Preference data collection is the silent engine behind every aligned model you've used. In large language models like GPT-4 or Claude, thousands of human annotators write prompts, generate multiple responses, and rank them. That data trains a reward model, which in turn drives RLHF. Without it, the model would just be a next-word predictor. In small language models that run on your phone, the same preference data — often distilled from larger models via synthetic data generation — teaches the small model to be harmless and helpful within its limited capacity. In retrieval-augmented generation (RAG) systems, preference data is collected by comparing entire retrieved-plus-answer pipelines: for the same question, one system returns a correct, relevant answer with proper citations, another produces a plausible but unsourced claim. Annotators prefer the former, and that signal optimises not only the generator but also the retriever's ranking. In agentic systems that plan and call tools, preference data is collected on multi-step trajectories: a sequence of tool calls, observations, and final answers. Pairwise comparisons of whole trajectories teach the agent to choose better plans. This is far more expensive to collect — annotators must verify tool outputs — but it's essential for teaching planning. In everyday applied ML — recommendation, fraud detection, forecasting — preference data shows up as user clicks (preferring one item over another), human reviews of flagged transactions, or expert forecasts compared. The common thread: whenever you need a model to rank, choose, or behave, you need paired comparisons. A developer uses libraries like `datasets`, `trl`, or custom annotation platforms (Labelbox, Scale AI) to manage collection. They write code to load prompts, sample multiple responses, send them to annotators, and parse their choices into a training set. What breaks when done badly: annotator disagreement and fatigue corrupt labels; off-policy data (from an older model) causes distribution drift; and if the data is too uniform, the reward model overfits and reward hacking appears. The Craft of Post-Training stresses that preference data quality dominates algorithm choice — no amount of clever optimisation can recover from garbage annotations.


---

Read this on the web: https://basavapurushottam.com/learn-ai/preference-data-collection
