# Alignment Case Study

A worked end-to-end alignment example — collect preferences, train a reward model, optimise the policy, then evaluate the result honestly.

**Rests on:** Preference Data Collection, Reward Modeling

*LLMs and Generative AI · Post-Training Alignment*


## Why learn this

This section matters because it reveals the hidden mechanics behind how AI assistants learn to speak like humans. You have likely interacted with a chatbot that gave a confident but incorrect answer, or one that was rude when it should have been helpful. This case study shows exactly how we fix those failures. It does not rely on abstract theory; it uses concrete steps: gathering real human opinions on which answers are better, turning those opinions into a scoring system, and then using that score to tweak the AI’s behavior. The specific benefit is that you will see how a simple idea—asking people to choose between two responses—can be scaled to make an AI safer and more useful in the real world.


## The intuition

Imagine you are teaching a child to write polite emails. You do not just hand them a dictionary. You watch them write an email. If they write 'Stop it', you show them a better version: 'Could you please stop?'. You point out that the second one is kinder. Over time, as they receive more examples of 'this phrasing is kind' versus 'that phrasing is rude', their writing improves. In the case study, we use specific numbers to make this concrete. Suppose we have a model that generates two possible responses to a question. Human judges look at Response A and Response B. If 7 out of 10 judges prefer Response A, we know Response A is 'better'. We then adjust the model’s internal settings slightly to make it more likely to generate Response A next time. This loop happens thousands of times with specific data points until the model consistently chooses the preferred style.


## An analogy

Think of this process as training a dog using positive reinforcement and a clicker. You want the dog (the AI) to sit (produce helpful text). You do not explain the physics of sitting. Instead, you wait for the dog to sit, and then you press a clicker (the reward model) and give a treat. The clicker predicts good behavior before the treat arrives. Over time, the dog learns that pressing its paws down triggers the clicker, so it sits faster and more reliably. The catch is that the clicker might get confused: if you accidentally click when the dog stands up to leave the room, the dog might start sitting less often because it thinks standing up is what gets the treat. This misunderstanding of what specifically triggered the reward is the core challenge we manage in alignment. This analogy holds until you realize the 'dog' is a massive statistical model calculating probabilities, not an animal feeling desire.


## Definition

Alignment case study refers to the practical process of improving a language model's behavior by collecting human judgments on its responses, training a scoring system on those judgments, and using that score to guide the model toward producing more helpful and harmless text, while monitoring for unintended side effects.


## How it connects

This case study sits alongside 'Reward Modeling', which is the technique of building the scoring system from human preferences, and 'Direct Preference Optimization', which is a method for updating the model directly based on those preferences without needing an intermediate scoring step. It also connects to 'Reward Hacking', which describes situations where the AI learns to maximize its score by cheating rather than actually improving its behavior.


## The mathematics

This case study rests on a branch of mathematics called probability — specifically, how we turn human choices into numbers we can optimise. The key idea is that we never have a perfect score for 'how good is this response?' Instead, we have a human or an AI telling us 'this response is better than that one'. The Bradley-Terry model takes these pairwise comparisons and converts them into a single number for each response, called a 'score' or 'reward'. The rule is simple: if response A is preferred over B, then the score of A should be higher than the score of B. The more often A is preferred, the higher its score relative to B. We then train a model to predict these scores for any new response. The mathematics we need is just arithmetic: adding, subtracting, and dividing to update scores. We do not need calculus at this stage, but you will see later that gradient descent (which uses calculus) is what actually optimises the policy. For now, focus on the arithmetic of turning comparisons into scores.

The Bradley-Terry model says that the probability of preferring A over B is:

P(A > B) = σ(r_A - r_B)

where σ is the sigmoid function (which squashes any number into a 0-1 range), and r_A, r_B are the scores for A and B. During training, we adjust the scores to maximise the probability of the observed comparisons.

In modern implementations, the scores come from a neural network called a reward model, and the update is done via gradient descent on the loss:

L = -log(σ(r_A - r_B)) for a comparison where A is preferred.


## A worked example

```
We have four responses to a prompt. We asked a human annotator to compare them pairwise. They said: A > B, A > C, B > D, C > D, and B > C (where '>' means preferred). We will assign an initial score of 0 to each, then update them using the Bradley-Terry update rule.

Step 1: Start: scores A=0, B=0, C=0, D=0   # We initialise all scores to zero.
Step 2: For comparison A>B: A gets +1, B gets -1 (simple version)   # A simple update: give +1 to the winner, -1 to the loser. This is a very basic rule, but it works to illustrate the idea. Real Bradley-Terry uses a softer update based on the probability of preference.
Step 3: After A>B: A=1, B=-1, C=0, D=0   # We move one point from B to A.
Step 4: After A>C: A=2, C=-1, B=-1, D=0   # Again, +1 to A, -1 to C.
Step 5: After B>D: B=0, D=-1, A=2, C=-1   # B gains, D loses.
Step 6: After C>D: C=0, D=-2, A=2, B=0   # C gains, D loses again.
Step 7: After B>C: B=1, C=-1, A=2, D=-2   # Final scores after all five comparisons.

Result: The final scores are: A=2, B=1, C=-1, D=-2. This matches the preference order we observed (A > B > C > D). In practice, the scores would be normalised and have a more sophisticated update, but this arithmetic gives you the core idea.
```


## From scratch, in Python

Let's write a simple script that simulates the whole alignment pipeline: given a dataset of preferences, it trains a tiny reward model (a linear model), then uses that reward model to score new responses and select the best one.

First, we need a preference dataset. It will be a list of tuples: (prompt, response_a, response_b, preference) where preference is 1 if A is preferred, 0 if B is preferred.

```python
# Data: us = prompt (as a simple feature vector), a and b are two responses (also features)
# For illustration, we use 2-dimensional features, but in reality they would be embeddings.
preferences = [
    ([1.0, 2.0], [2.0, 1.0], 1),  # A preferred
    ([2.0, 1.0], [1.0, 3.0], 0),  # B preferred
    ([0.5, 0.5], [1.5, 0.0], 1),  # A preferred
    ([1.0, 3.0], [2.0, 2.0], 0)   # B preferred
]

# Reward model: we use a simple linear function r(features) = w1*f1 + w2*f2 + b
# We'll train it with a simple gradient descent.

weights = [0.0, 0.0]
bias = 0.0
learning_rate = 0.1

def reward(features):
    return weights[0]*features[0] + weights[1]*features[1] + bias

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

import math

# Train the reward model: for each preference pair, update weights to increase the reward difference.
for epoch in range(100):
    for prompt, a, b, pref in preferences:
        r_a = reward(a)
        r_b = reward(b)
        if pref == 1:
            # want r_a > r_b, so move weights to increase r_a - r_b
            # simple update: add learning_rate * (1 - sigmoid(r_a - r_b)) * (a - b) to weights
            diff = a[0] - b[0], a[1] - b[1]
            err = 1 - sigmoid(r_a - r_b)
            weights[0] += learning_rate * err * diff[0]
            weights[1] += learning_rate * err * diff[1]
            bias += learning_rate * err
        else:
            # want r_b > r_a, so move weights to increase r_b - r_a
            diff = b[0] - a[0], b[1] - a[1]
            err = 1 - sigmoid(r_b - r_a)
            weights[0] += learning_rate * err * diff[0]
            weights[1] += learning_rate * err * diff[1]
            bias += learning_rate * err

print("Trained reward weights:", weights, "bias:", bias)

# Now we optimise the policy: we generate new responses (here we just search over a small set) and pick the one with highest reward.
# In reality, the policy generates responses and we use RL to update it, but here we just choose the best from candidates.

def generate_candidates():
    # For simplicity, we have a fixed set of candidate responses as feature vectors.
    return [ [1.0, 1.0], [2.0, 2.0], [0.5, 2.5], [1.5, 1.0] ]

candidates = generate_candidates()
scores = [reward(c) for c in candidates]
best_idx = max(range(len(scores)), key=lambda i: scores[i])
print("Best response (by reward):", candidates[best_idx], "with score", scores[best_idx])

# Evaluation: we have a held-out set of preferences to test how well the reward model predicts.
# This is a separate step in the pipeline.
```


## The same thing, with a framework

In PyTorch, you would define a neural network as the reward model and use a DataLoader. The core idea is the same: you have a dataset, you define a loss function (e.g., BCEWithLogitsLoss on the difference), and you optimise with an Adam optimizer. Here is a sketch:

```python
import torch
import torch.nn as nn
import torch.optim as optim

class RewardModel(nn.Module):
    def __init__(self, input_dim):
        super().__init__()
        self.fc = nn.Linear(input_dim, 1)

    def forward(self, x):
        return self.fc(x).squeeze(-1)

model = RewardModel(input_dim=2)
optimizer = optim.Adam(model.parameters(), lr=0.01)
loss_fn = nn.BCEWithLogitsLoss()

# Pseudo-training loop
for epoch in range(100):
    for batch in dataloader:
        prompts, a, b, pref = batch
        r_a = model(a)
        r_b = model(b)
        logits = r_a - r_b  # shape (batch,)
        loss = loss_fn(logits, pref.float())
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
```

Underneath, it is doing exactly the same thing as our from-scratch loop, but with matrices and automatic differentiation. The reward model is a small MLP, and the preference loss is logistic regression on the difference.


## What the code does

Let's go through the from-scratch code. We have a small list of preferences. Each tuple has a prompt (unused here) and two responses as feature vectors, and a preference label. We initialise weights and bias to zero. The `reward` function computes a weighted sum. The training loop is the heart: for each preference, we compute the rewards for a and b. If a is preferred, we want r_a to be much higher than r_b. The `err` term is how much we are wrong: if sigmoid(r_a-r_b) is 1 (perfect), err is 0 and we update nothing. If it is 0 (totally wrong), err is 1 and we update a lot. The update moves the weights in the direction of the feature difference. This is just a manual gradient descent. After training, we use the reward model to pick the best candidate from a small set. Finally, in a real pipeline, you would evaluate on holdout data to see if the reward model generalises, and also check how the policy (the language model) changed.


## Common misconceptions

**Reward model knows what 'good' means** — The reward model understands what makes a good response, so it can judge anything correctly.

The reward model is a statistical function trained on a finite set of human preferences. It has no true understanding. It just learns to approximate patterns in data. In fact: A reward model is a compressed representation of what annotators preferred on the training data. It can be wrong on new inputs, which is why we need to evaluate and monitor.

**Preferences are objective** — Collecting preferences is just a matter of asking people which they like, and that gives a perfect signal.

Annotators disagree, are influenced by factors like order, verbosity, and confidence, and can be inconsistent, especially when tired. The annotation process is subjective and noisy. In fact: Preference collection is a research problem. You need careful guidelines, multiple annotators, and checks on agreement to build a reliable dataset.

**Aligning makes the model perfect** — After alignment, the model will always be helpful and harmless, and never make mistakes.

Alignment only steers the model toward the reward signal, which is imperfect. There are trade-offs: safety vs. helpfulness, and the model can still be wrong or find loopholes. In fact: Alignment changes the distribution of outputs, but does not guarantee correctness. Evaluation on a variety of tests is necessary to find failures.

**Optimising the reward is the same as optimising the intent** — If we maximise the reward model's score, we maximise the true quality of the response.

This is the core of reward hacking. The reward model is a proxy, not the real thing. Maximising a proxy can exploit its blind spots. In fact: We optimise the reward model, but we must be aware that it is a proxy. The goal is to align with human values, not just with the reward model.

**More preference data is always better** — If we gather a huge amount of preference data, the reward model will automatically be great.

Data quality and annotation consistency matter more than volume. If the data is noisy or biased, more of it just amplifies the problem. In fact: Invest in clean guidelines, do regular audits, and consider using smaller, high-quality datasets over massive, messy ones.


## In short

The alignment case study is the full journey from raw human preferences to a tuned model. It involves collecting preference pairs, turning them into a numeric reward model using the Bradley-Terry model, optimising a language model policy against that reward, and rigorously evaluating both the intended improvement and any collateral damage. The core insight is that the reward model is a proxy, and optimising a proxy opens the door to reward hacking, so honest evaluation is not optional — it is the only safety net.


## Where this is used

Now, where does this case study show up in real AI systems? It's everywhere in post-training. Let's be specific.

**Large Language Models (like ChatGPT, Claude, Gemini).** After pretraining and supervised fine-tuning, these companies collect massive preference datasets — either from human raters or from AI 'judges' (often a bigger model). They train a reward model on this data. Then they use an algorithm like PPO (Proximal Policy Optimization) or DPO to update the language model so that it produces responses that score high on the reward model. The mechanism: the language model is the policy. It generates responses to prompts, the reward model scores them, and the policy is updated with gradient ascent to increase that score. Training happens over thousands of prompts, on thousands of GPUs, for weeks. A misstep in the reward model — for example, rewarding long-winded but unhelpful answers — leads to reward hacking, where the model becomes verbose to game the score.

**Small Language Models (on phones/laptops).** In a tiny model running on a Raspberry Pi or phone, you do not have room for a separate reward model and a heavy RL loop. Instead, you often use DPO, which directly optimises from preference pairs without a learned reward model. The case study is simpler: you still need preference data, but you skip the reward model training and just update the small policy directly. The preference dataset is often distilled from a larger model. The risk: the small model might learn to mimic the large model's style but with more errors, so evaluation is crucial.

**Retrieval-Augmented Generation (RAG).** RAG systems fetch information from a database to ground the answer. Alignment applies here too: you might collect preferences on whether the answer is faithful to the retrieved context. A reward model can be trained to score not just the final response but also whether it uses the context correctly. The policy is the generator, but the reward model also has to account for the retrieved documents. In practice, a developer writes a reward model that takes the prompt, the retrieved documents, and the response, and scores them. If the reward model ignores the context, the model learns to make up facts that sound good.

**Agentic Systems (planning, tool calls).** When an AI agent decides to call a tool or execute an action, the reward is not just about text — it is about whether the action succeeded. An alignment case study here means collecting trajectories (sequences of actions) and comparing them. The reward model is trained on these trajectories, scoring whole chains of actions rather than single responses. Then the policy is optimised with an algorithm like GRPO, which samples many possible action sequences and updates based on which ones led to higher rewards. The full pipeline is similar: data collection (maybe by letting the agent act and asking humans to rank the outcomes), reward modelling, policy optimisation.

**Everyday Applied ML (recommendation, fraud detection).** In recommendation systems, you have implicit preferences: clicks, watch time, purchases. The case study translates: each item is a response, and you build a reward model that predicts how likely a user is to interact. Then you optimise the recommender to maximise that reward. In fraud detection, you have labels of final outcome (fraud or not) but not step-by-step preferences; you might prefer sequences of rules, and use a reward model to guide the investigation agent. The developer writes the reward model as a classifier and uses it to score candidate actions, then uses a policy optimisation or a simple ranking loss to update the system.

**What a developer actually does:** They write scripts to collect and clean preference data (often using LLM-as-a-judge with guardrails). They define a reward model in PyTorch (like the sketch above) and train it with a loss on preference pairs. Then they use a library like TRL (Transformer Reinforcement Learning) to run PPO or DPO on their base model. They also write evaluation harnesses (like LightEval or Promptfoo) to test the final model on standard benchmarks and also on specific adversarial prompts. If evaluation reveals regressions (e.g., the model is safer but less helpful), they tweak the data or the training hyperparameters.

**What breaks when done badly:** The reward model overfits to the annotation idiosyncrasies, and the policy learns to produce "reward-maximising" nonsense. You see this in chatbots that produce overly long, sycophantic, or incorrect answers because those were rewarded. Evaluation is often done on the same metrics used for training, so the model looks good on the benchmark but fails in real use. The lesson from the library: 'The gap between paper method and running pipeline is where the work is' — you have to build the infrastructure for data, training, and evaluation, and that is the hard part.


---

Read this on the web: https://basavapurushottam.com/learn-ai/alignment-case-study
