# Reward Modeling

Training a model to predict human preference, producing a scalar signal that stands in for human judgement during optimisation.

**Rests on:** Bradley-Terry Model, Preference Data Collection

**Leads to:** Alignment Case Study, Direct Preference Optimization, GRPO, Reward Hacking

*LLMs and Generative AI · Post-Training Alignment*


## Why learn this

Reward modeling is the mechanism that allows artificial intelligence systems to distinguish between good answers and bad ones. Without it, a system would simply generate text without knowing which version was most helpful, honest, or safe for you. This concept explains how assistants like ChatGPT learn to avoid harmful content and prefer useful responses. It also underlies recommendation systems that tailor content to your tastes and autonomous agents that decide which tools to use. By understanding this, you see how we turn vague human values into precise signals that guide a computer’s behavior.


## The intuition

Imagine you are training a puppy. You do not want to give it a treat for every single action because you cannot be there all the time to judge each move. Instead, you might enlist a friend to watch the puppy and give a score from 0 to 10. That number tells the trainer whether the action was good or bad. Reward modeling works exactly this way: we train a separate system to act as that 'judge.' This judge looks at a response and produces a single number, called a reward, which indicates quality. The main system—the one generating the text—is the puppy. The judge is not the final goal; it is a stand-in for human judgment because asking humans to grade every possible response would be too slow and expensive. So, we collect examples where humans have said, 'Response A is better than Response B.' We train our judge model on these comparisons. Once trained, this judge can instantly score any new response, allowing us to train the main system quickly and at scale.


## An analogy

Think of a movie critic. You are the filmmaker, trying to make a movie that audiences love. You cannot screen the rough cut for every person on Earth. So, you hire a critic who has watched thousands of movies and knows what makes a good one. You show the critic your work, and they give you a score out of 10. You use that score to decide which scenes to keep and which to cut. The critic is not the audience, but they are a close enough approximation to guide your decisions. This works well, but it has one flaw: the critic can be fooled. If you know the critic loves car chases, you might add a car chase to every scene, even if it ruins the story. This is called 'reward hacking'—optimizing for the critic’s score rather than the audience’s true enjoyment. The judge model is a compressed summary of what humans preferred, and like any compression, it loses some detail. It optimizes that summary, not the original intent perfectly.


## Definition

Reward modeling is the process of training a system to predict which of two or more responses humans prefer, outputting a single number that guides the improvement of the main text-generating system.


## How it connects

This concept sits within post-training alignment, which refers to all adjustments made after a model has learned general language skills to make it useful and well-behaved. It connects directly to supervised fine-tuning, where a model learns by mimicking high-quality examples provided by humans, and preference optimization, which adjusts the model based on judgments of quality rather than direct examples. While reward modeling explicitly trains a judge to score responses, other methods like Direct Preference Optimization skip the explicit judge and use human preferences directly to guide the training signal.


## The mathematics

Reward modeling is a way of teaching a computer what humans like by showing it examples of choices. We do not tell the computer exactly how to calculate 'goodness.' Instead, we show it pairs of responses and tell it which one was better. The mathematics used to turn these 'better/worse' judgments into numbers comes from statistics, specifically a method called the Bradley-Terry model.

Here is how the pieces fit together, step by step.

1. The Score Function
We need a way to give every response a number. Let us call this number the 'reward.' We use a function, which we will name r. When you put a response x into this function, it outputs a single real number (a positive or negative decimal).

r(x) = the reward score for response x.

This function r is what we are trying to learn. It has internal settings called 'weights.' At the start, these weights are random, so the scores are meaningless. We will adjust them later.

2. Comparing Two Responses
In our training data, we have pairs of responses for the same prompt. One is the 'chosen' response (the one humans liked) and one is the 'rejected' response (the one humans did not like).

We want the model to assign a higher score to the chosen response than to the rejected one. Let us define the difference between these scores:

difference = r(chosen) - r(rejected)

If the chosen response is truly better, this difference should be a positive number. If the model is confused and gives the rejected response a higher score, this difference will be negative.

3. Turning the Difference into a Probability
A raw difference (like 0.4 or -2.5) is hard to interpret as a 'chance' of being correct. We need to squash this number into a value between 0 and 1, which represents a probability. We use a specific curve for this called the logistic function, or sigmoid function.

The formula for the sigmoid function is:
sigmoid(z) = 1 divided by (1 + e raised to the power of -z)

Here, z is the difference we calculated above. The symbol 'e' is a constant number approximately equal to 2.718. This function has a special property: if z is a large positive number, the result is close to 1. If z is a large negative number, the result is close to 0. If z is 0, the result is exactly 0.5.

So, the probability that the chosen response is better, given our current model, is:
P(chosen is better) = sigmoid(r(chosen) - r(rejected))

4. Measuring the Error (The Loss)
We want this probability to be as close to 1 as possible. If the model is perfect, it says 'I am 100% sure A is better than B.' If the model is wrong, it might say 'I think A is better, but only with 51% confidence.'

To measure how bad the model's guess is, we use a 'loss' function. The standard loss for this problem is called the negative log-likelihood loss. In plain terms, it works like this:

Loss = - (the natural logarithm of P(chosen is better))

Why do we use the logarithm? Because probabilities are small numbers (like 0.6). Multiplying many small probabilities together makes them tiny and hard to work with mathematically. The logarithm turns multiplication into addition, which is much easier for computers to optimize. The negative sign ensures that when the probability is high (close to 1), the loss is low (close to 0). When the probability is low (close to 0), the loss becomes very large.

So the full formula for the loss on one pair is:
Loss = - log( sigmoid(r(chosen) - r(rejected)) )

5. Training the Model
We have many such pairs in our dataset. We calculate the loss for each pair and add them up (or average them). This total number tells us how wrong our model is overall. We then use an algorithm called gradient descent to adjust the weights inside the function r. The goal of gradient descent is to find new weights that make this total loss smaller. We repeat this process many times until the loss stops decreasing significantly.


## A worked example

Let us walk through a single example with real numbers.

The Setup:
Prompt: 'What is the capital of France?'
Response A (Chosen): 'Paris.'
Response B (Rejected): 'I think it's somewhere in Europe, maybe Lyon?'

We start with a reward model that has not learned anything yet. It assigns arbitrary scores.
Initial score for A: r(A) = 1.2
Initial score for B: r(B) = 0.8

Step 1: Calculate the difference in scores.
The Bradley-Terry model relies on the gap between the two scores.
Difference = r(A) - r(B)
Difference = 1.2 - 0.8 = 0.4

This positive number (0.4) tells us that our current model thinks A is better than B, but not by much. If the difference were negative, the model would think B is better.

Step 2: Convert the difference into a probability using the sigmoid function.
We need to find sigmoid(0.4). The formula is:
sigmoid(z) = 1 / (1 + e^(-z))

First, calculate the exponent part: e raised to the power of -0.4.
e^(-0.4) ≈ 0.6703

Next, add 1 to this result:
1 + 0.6703 = 1.6703

Finally, divide 1 by this sum:
1 / 1.6703 ≈ 0.5987

So, the model predicts that there is a 59.87% chance that A is better than B. This is not very confident. We want this number to be close to 1.0 (100%).

Step 3: Calculate the Loss.
The loss measures how far we are from the ideal outcome (probability = 1). The formula is:
Loss = - log(sigmoid(0.4))

First, take the natural logarithm of the probability we just found:
log(0.5987) ≈ -0.5128

Then, apply the negative sign:
Loss = - (-0.5128) = 0.5128

A loss of 0.5128 is our current error score for this single pair.

Step 4: Update the model weights (Conceptual Step).
The computer uses the loss to figure out which way to change the weights inside r(A) and r(B). It wants to increase r(A) and decrease r(B) so that the difference becomes larger. Let us imagine one step of this update process results in new scores:
New score for A: 1.5
New score for B: 0.5

Step 5: Recalculate with the new scores.
Difference = 1.5 - 0.5 = 1.0

Now, plug 1.0 into the sigmoid function:
sigmoid(1.0) = 1 / (1 + e^(-1.0))
e^(-1.0) ≈ 0.3679
1 + 0.3679 = 1.3679
1 / 1.3679 ≈ 0.7311

The model now predicts a 73.11% chance that A is better. This is more confident than before.

Calculate the new loss:
Loss = - log(0.7311)
log(0.7311) ≈ -0.3132
Loss = 0.3132

Comparison:
The old loss was 0.5128. The new loss is 0.3132. The loss has gone down. This means the model's guess is closer to the truth. We repeat this process thousands of times with many different prompts until the loss is as small as possible.


## From scratch, in Python

```python
import math

# We'll use a very simple reward model: a function that assigns a score based on a few features.
# For this example, let's say the reward model is a linear function: score = w1*len(response) + w2*word_count_positive - bias
# We'll train it with gradient descent on one preference pair.

# Initialize parameters (like weights and bias)
w_length = 0.1
w_positive = 0.5
bias = 0.2

# Training data: (chosen, rejected) pairs as lists of features
# Feature 1: length of response (number of characters)
# Feature 2: number of positive words (like 'good', 'great', 'yes')
chosen = [5, 2]  # short response, some positive words
rejected = [10, 0]  # long response, no positive words

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

def score(response, w_len, w_pos, b):
    return w_len * response[0] + w_pos * response[1] - b

# Training loop (just one step for illustration)
learning_rate = 0.1

# Forward pass: compute scores
s_chosen = score(chosen, w_length, w_positive, bias)
s_rejected = score(rejected, w_length, w_positive, bias)

# Compute loss
diff = s_chosen - s_rejected
loss = -math.log(sigmoid(diff))

# Gradient descent (update each parameter)
# gradient for w_length: (sigmoid(diff) - 1) * (chosen[0] - rejected[0])
# Similarly for others
grad_w_length = (sigmoid(diff) - 1) * (chosen[0] - rejected[0])
grad_w_positive = (sigmoid(diff) - 1) * (chosen[1] - rejected[1])
grad_bias = -(sigmoid(diff) - 1) * 1  # since bias is subtracted, gradient is negative

w_length -= learning_rate * grad_w_length
w_positive -= learning_rate * grad_w_positive
bias -= learning_rate * grad_bias

print(f"Updated parameters: w_length={w_length:.2f}, w_positive={w_positive:.2f}, bias={bias:.2f}")
print(f"New loss: {-math.log(sigmoid(score(chosen, w_length, w_positive, bias) - score(rejected, w_length, w_positive, bias))):.4f}")
```


## The same thing, with a framework

```python
In PyTorch, you would use a neural network as the reward model and a built-in loss function like BCEWithLogitsLoss. Here's a sketch:

import torch
import torch.nn as nn

# Reward model: a simple MLP
reward_model = nn.Sequential(
    nn.Linear(2, 4),
    nn.ReLU(),
    nn.Linear(4, 1)
)

# Optimizer
optimizer = torch.optim.Adam(reward_model.parameters(), lr=0.001)

# Data: features for chosen and rejected
chosen_features = torch.tensor([5.0, 2.0])
rejected_features = torch.tensor([10.0, 0.0])

# Forward pass
s_chosen = reward_model(chosen_features)
s_rejected = reward_model(rejected_features)

# Loss: negative log sigmoid of difference
loss = -torch.log(torch.sigmoid(s_chosen - s_rejected))

# Backprop
optimizer.zero_grad()
loss.backward()
optimizer.step()

This is what happens under the hood: the model computes the scores, the loss measures how well the chosen is ranked above the rejected, and backpropagation updates the weights to reduce that loss.
```


## What the code does

The from-scratch code defines a simple reward model using a linear function. It uses gradient descent to adjust the weights based on a single preference pair. The sigmoid function converts the score difference to a probability, and the loss is the negative log of that probability. Minimizing this loss pushes the chosen score up and the rejected score down. The PyTorch version does the same but with a neural network and automatic differentiation.


## Common misconceptions

**Reward model is the final goal** — The reward model itself is the product I want to ship.

The reward model is a proxy for human judgment, not the ultimate objective. It is a means to train the policy model. The end user interacts with the policy, not the reward model. In fact: The reward model is used only during training to provide a scalar signal. The final deployed system is the policy model, which has been shaped by that signal.

**Reward model is perfect** — If I train a reward model well, it will accurately represent what humans want, so the policy will be perfect.

Reward models are approximations. They are trained on a finite set of preferences, so they can miss nuances or be fooled. Reward hacking occurs when the policy exploits these imperfections. In fact: Reward models are fallible. They are a compressed representation of human preferences, and like any compression, they lose information. The policy can exploit that loss.

**Reward model is the same as the policy** — The reward model and the policy model are the same thing — they both generate responses.

The reward model gives a score, not a response. The policy generates responses. They are separate models with different tasks. In fact: The reward model outputs a single number (the reward) for any given response. The policy is the generative model that produces responses. They work together: the policy generates, the reward model scores, and the policy learns to generate higher-scoring responses.

**Reward model judges absolute quality** — The reward model's score is an absolute measure of how good a response is.

Reward models are trained on pairwise comparisons, so they only learn relative ordering. A score of 5 on one task does not mean better than a score of 3 on another task unless they were compared. In fact: Reward model scores are relative, not absolute. They are only meaningful in the context of the distribution they were trained on, comparing two or more responses to the same prompt.

**Reward model eliminates the need for human feedback** — Once I train a reward model, I never need humans again.

The reward model is trained on human feedback, and its quality depends on that data. As the policy changes, the reward model may become outdated, and human feedback may be needed to re-train it. In fact: Reward models are only as good as the data they are trained on. Human feedback is still needed to evaluate the reward model's accuracy and to collect new preferences when the distribution shifts.

**Higher reward score means better response for the user** — A response with a higher reward score is always the one a human would prefer.

The reward model can be hacked. The policy may find ways to game the score, for example, by making responses longer or more confident, even if they are not actually better. In fact: Reward hacking is a real phenomenon. The policy optimizes the reward model's score, not the underlying intent. It is important to evaluate the policy with held-out human preferences, not just the reward score.


## In short

Reward modeling trains a separate model to predict human preferences, outputting a scalar score. This score is used to guide the optimization of the policy model during RLHF. The key is that the reward model is a compressed, fallible proxy for human judgment. It fails separately from the policy, and its errors become the policy's optimization target, leading to reward hacking. The Bradley-Terry model is the mathematical bridge from pairwise preferences to scores. In practice, reward models are used in LLMs, RAG, agents, and recommendations. The code shows a simple implementation and the library version with PyTorch.


## Where this is used

Reward models are used everywhere in modern AI. In large language models (LLMs), they are the core of RLHF: after supervised fine-tuning, a reward model is trained on human preferences, and then the LLM is fine-tuned to maximize the reward model's score. This shapes the model's tone, helpfulness, and harmlessness. For example, ChatGPT's helpfulness was tuned this way. In smaller models that run on phones or laptops, reward models are used for on-device preference optimization, but often with efficient techniques like LoRA to keep memory low. The reward model itself might be a small transformer that runs quickly on-device. In retrieval-augmented generation (RAG), reward models can rank retrieved passages: they are trained to prefer relevant passages over irrelevant ones, and then used to re-rank search results before feeding them to the LLM. In agentic systems that plan and call tools, reward models are used to evaluate the quality of each action. The agent proposes a sequence of tool calls, and a reward model scores each step, guiding the agent to choose actions that lead to a successful outcome. In everyday applied ML, reward models are used in recommendation systems to rank items based on user preferences, and in fraud detection to score suspicious activity. A developer implementing reward modeling in practice would write code to: (1) collect preference data, (2) train a reward model using a library like PyTorch or TensorFlow, (3) use the reward model in an RL loop with a library like HuggingFace TRL, which provides RewardTrainer and PPOTrainer classes. When done badly, a common failure is reward hacking, where the policy finds a way to score high without actually satisfying the intent.


---

Read this on the web: https://basavapurushottam.com/learn-ai/reward-modeling
