# Direct Preference Optimization

Optimising directly on preference pairs without training a separate reward model, using a loss derived from the RLHF objective.

**Rests on:** Instruction Tuning, Reward Modeling

**Leads to:** ORPO and Preference Optimization

*LLMs and Generative AI · Post-Training Alignment*


## Why learn this

This page explains how Direct Preference Optimization lets you teach an AI to give better answers by showing it examples of good and bad responses directly, skipping the complex step of training a separate 'judge' model. By the end, you will understand exactly how this direct method changes the training process and why it is often simpler and more stable than older approaches that relied on separate reward scoring or Reinforcement Learning from Human Feedback.


## The intuition

Imagine you have a language model like a chatbot. It can respond to a prompt in many different ways, but not all are equally good. Humans can compare two answers and say 'this one is better' or 'this one is worse'. This comparison is called a 'preference'. The traditional way (used in Reinforcement Learning from Human Feedback) to teach a model these preferences is to first create a separate 'reward model'. A reward model is another AI that has been trained to look at a response and give it a score, like a judge. You'd train this judge to predict which answer a human would prefer. Then, you'd use a complex process (reinforcement learning) to tweak your chatbot's responses so they get higher scores from this judge. That's a lot of moving parts and can be tricky. 'Direct Preference Optimization' (DPO) takes a shortcut. It says: 'What if we skip the judge entirely and update the chatbot directly based on human preferences?' That's the core idea. Instead of learning a separate scoring rule and then following it, DPO recalculates the 'best' response direction directly from the paired examples of good and bad answers. It's a simpler, more direct route to the same goal.


## An analogy

Think of teaching a child to write better essays. The traditional (Reinforcement Learning from Human Feedback) approach would be: 1) Hire a professional essay grader to give a score to every essay. 2) Have the child write many versions and try to adjust their writing to get a higher score from this grader. This is slow, requires a very good grader, and the child might just learn to game the grader's specific preferences. DPO is a different strategy. Instead of a grader, you give the child many pairs of essays on the same topic: one deemed excellent and one deemed poor. The child's job is to simply adjust their writing style to be more like the good essays and less like the bad ones. They don't need a grader to give them a numeric score; they just need the direct comparison. This analogy breaks down a little because AI models aren't 'understanding' like a child. The child infers a hidden rule, while DPO calculates a precise mathematical adjustment to the model's internal parameters.


## Definition

Direct Preference Optimization is a method for training an AI model to better align with human preferences by directly adjusting its response generation rule using pairs of chosen and rejected examples, without needing a separate reward model. This approach optimizes the likelihood of good answers relative to bad ones in a single step.


## How it connects

You are learning this from a clean slate, so we'll build it up from nothing. DPO sits inside the world of 'Post-Training Alignment', which is everything you do to a large language model after its initial pre-training to make it helpful, safe, and aligned with what people want. It is often discussed alongside 'Supervised Fine-Tuning' — where you first teach the model on examples of good behavior using labeled data — and it's a direct alternative to the multi-step pipeline that combines 'Reward Modeling' (training a separate AI to score responses) and 'Reinforcement Learning from Human Feedback' (using those scores to guide learning). It also connects closely with other methods like ORPO, which is another way to optimize directly from preferences, and with statistical frameworks like the Bradley-Terry Model, which provides the mathematical bridge from human comparisons to a useful score. The quality of your preference data, as noted in 'Preference Data Collection', is the bedrock of all of these techniques.


## The mathematics

**Probability and Statistics: The Language of Uncertainty**

This concept rests on **probability and statistics**. This is the branch of mathematics used to measure how likely an event is to happen when we do not know the outcome with certainty. In Direct Preference Optimization (DPO), we use probability because human opinions are never perfectly consistent. We cannot simply say "Response A is correct." Instead, we say "Response A is more likely to be preferred than Response B."

**How the Model Assigns Probabilities**

Think of the AI model as having a large internal list of every possible response it could give to a prompt. For any specific prompt, the model assigns a number between 0 and 1 to each possible response. This number is called the **probability**. 

- If the probability is close to 1, the model thinks that response is very likely.
- If the probability is close to 0, the model thinks that response is very unlikely.
- The sum of the probabilities for all possible responses must equal exactly 1.

We write this as **pi-sub-theta** (read "pi of theta") of **y-given-x**. Here:
- **theta** (theta) represents the current settings or "weights" of the model. These are the numbers that define how the model thinks.
- **x** is the prompt given to the model.
- **y** is a specific response.
- **pi-sub-theta(y | x)** means "the probability that the model with settings theta produces response y, given prompt x."

**The Goal: Comparing Two Responses**

In DPO, we do not train the model on single responses. We train it on pairs of responses for the same prompt:
1. **y-sub-w** (read "y-sub-w" or "chosen response"): The response humans liked.
2. **y-sub-l** (read "y-sub-l" or "rejected response"): The response humans disliked.

Our goal is to adjust the model's settings (**theta**) so that **pi-sub-theta(y-sub-w | x)** becomes larger and **pi-sub-theta(y-sub-l | x)** becomes smaller. We want the ratio of these two probabilities to be as large as possible.

**The Ratio and the Logarithm**

To compare how much better the chosen response is, we look at the ratio:

Ratio = pi-sub-theta(y-sub-w | x) divided by pi-sub-theta(y-sub-l | x)

If the model prefers the chosen response, this ratio will be greater than 1. If it prefers the rejected response, the ratio will be less than 1.

Working directly with ratios can be difficult because they grow very fast. So, we take the **natural logarithm** (often written as **log**) of the ratio. The logarithm turns multiplication into addition and keeps numbers smaller and easier to handle. 

Log-Ratio = log(pi-sub-theta(y-sub-w | x)) minus log(pi-sub-theta(y-sub-l | x))

This Log-Ratio tells us how much more likely the chosen response is than the rejected one, on a logarithmic scale.

**The Sigmoid Function: Squashing to 0-1**

We want a single number that represents our "confidence" that the model prefers the chosen response. This number must be between 0 and 1. We use a function called **sigmoid** (read "sig-moid") to convert the Log-Ratio into this confidence score.

The sigmoid function takes any real number (positive or negative) and squashes it into the range (0, 1). 
- If the Log-Ratio is very large and positive, sigmoid returns a number close to 1 (high confidence).
- If the Log-Ratio is very large and negative, sigmoid returns a number close to 0 (low confidence).
- If the Log-Ratio is 0, sigmoid returns exactly 0.5.

The formula for sigmoid of a number **z** is:
Sigmoid(z) = 1 divided by (1 + e raised to the power of -z)
Here, **e** is Euler's number, approximately 2.718.

**The DPO Loss Function**

We want the confidence score to be as close to 1 as possible for every pair we see. To measure how far we are from this goal, we use a **loss function**. The loss is a number that tells us how "wrong" the model is. We want to minimize this loss.

The DPO loss for one pair is:

L-sub-DPO = -1 times log(Sigmoid(beta times Log-Ratio))

Let's break down the parts:
- **beta** (beta) is a small positive number called the "temperature" or "strength" parameter. It controls how strongly we want to push the model toward the preference. A larger beta means we force the model to care more about the difference between chosen and rejected.
- **Log-Ratio** is the difference in log-probabilities calculated above.
- **Sigmoid(beta times Log-Ratio)** converts this scaled difference into a confidence score between 0 and 1.
- **log(Sigmoid(...))** takes the logarithm of that confidence. Since the confidence is less than 1, its log is negative.
- **-1 times ...** flips the sign so the loss is positive. 

If the model is perfect (confidence is 1), log(1) is 0, and the loss is 0. If the model is wrong (confidence is close to 0), log(close to 0) is a large negative number, and -1 times that is a large positive loss.

**The Full Formula**

In practice, we have many pairs of data, collected in a set **D**. We calculate the average loss over all these pairs. This is written as an **expectation**, denoted by **E-sub-D** (read "expectation over D").

L-sub-DPO(pi-sub-theta; pi-sub-ref) = E-sub-(x, y-sub-w, y-sub-l) from D [ -log(Sigmoid(beta * (log(pi-sub-theta(y-sub-w|x)) - log(pi-sub-theta(y-sub-l|x)) - log(pi-sub-ref(y-sub-w|x)) + log(pi-sub-ref(y-sub-l|x)))) ]

Wait, there is a crucial detail often missed. The formula above assumes we only look at the current model **pi-sub-theta**. But DPO compares the current model to a **reference model** (**pi-sub-ref**). The reference model is the original model before training started. We subtract the log-probabilities of the reference model to prevent the new model from drifting too far away from the original, which would make it unstable or nonsensical.

So the full Log-Ratio used in DPO is actually:
Log-Ratio = [log(pi-sub-theta(y-sub-w|x)) - log(pi-sub-ref(y-sub-w|x))] minus [log(pi-sub-theta(y-sub-l|x)) - log(pi-sub-ref(y-sub-l|x))]

This simplifies to:
Log-Ratio = log( pi-sub-theta(y-sub-w|x) / pi-sub-ref(y-sub-w|x) ) minus log( pi-sub-theta(y-sub-l|x) / pi-sub-ref(y-sub-l|x) )

So the final loss is:
L-sub-DPO = -E [ log(Sigmoid(beta * Log-Ratio)) ]


## A worked example

Let us walk through a simplified numerical example. We will ignore the reference model subtraction for this step to show the core mechanism, but note that in real DPO, you would subtract the reference probabilities first.

**Setup:**
- Vocabulary: Three possible responses: A, B, C.
- Prompt: "What is 2+2?"
- Human Preference: Chosen = A ("4"), Rejected = B ("5").
- Parameter **beta**: Let us set beta = 1 for simplicity.

**Step 1: Initial Probabilities**
Before training, the model is random. It assigns equal probability to all responses.
- P(A) = 0.33
- P(B) = 0.33
- P(C) = 0.33
(Note: 0.33 + 0.33 + 0.33 = 0.99, which is approximately 1. We will treat them as exactly 1/3 for precision in the ratio, but use decimals for clarity.)

**Step 2: Calculate the Ratio of Probabilities**
We compare the chosen response A to the rejected response B.
Ratio = P(A) divided by P(B)
Ratio = 0.33 / 0.33
Ratio = 1.0

Interpretation: The model thinks A and B are equally likely. It has no preference.

**Step 3: Calculate the Log-Ratio**
Log-Ratio = log(Ratio)
Log-Ratio = log(1.0)
Log-Ratio = 0

Interpretation: The logarithm of 1 is always 0. This confirms there is no preference.

**Step 4: Apply Beta and Sigmoid**
First, multiply by beta:
Scaled Value = beta * Log-Ratio
Scaled Value = 1 * 0 = 0

Next, apply the sigmoid function to this scaled value:
Confidence = Sigmoid(0)
Confidence = 1 / (1 + e raised to the power of -0)
Since e^0 = 1:
Confidence = 1 / (1 + 1) = 1 / 2 = 0.5

Interpretation: The model's confidence that A is better than B is 50%. This is random guessing.

**Step 5: Calculate the Loss**
Loss = -1 * log(Confidence)
Loss = -1 * log(0.5)
Using a calculator, log(0.5) is approximately -0.693.
Loss = -1 * (-0.693) = 0.693

Interpretation: The loss is high (0.693). This tells the model: "You are wrong. You should prefer A over B more strongly."

**Step 6: After Training Update**
Suppose the training algorithm adjusts the model's internal weights so that it now prefers A more.
New Probabilities:
- P(A) = 0.5
- P(B) = 0.2
- P(C) = 0.3 (Remains to sum to 1)

Recalculate the Ratio:
Ratio = P(A) / P(B) = 0.5 / 0.2 = 2.5

Recalculate the Log-Ratio:
Log-Ratio = log(2.5)
log(2.5) is approximately 0.916.

Apply Beta (still 1):
Scaled Value = 1 * 0.916 = 0.916

Apply Sigmoid:
Confidence = 1 / (1 + e raised to the power of -0.916)
e raised to the power of -0.916 is approximately 0.40.
Confidence = 1 / (1 + 0.40) = 1 / 1.40 ≈ 0.714

Calculate New Loss:
Loss = -1 * log(0.714)
log(0.714) is approximately -0.336.
Loss = -1 * (-0.336) = 0.336

**Result:**
The loss dropped from 0.693 to 0.336. The model's confidence increased from 0.5 to 0.714. This shows the model is learning to prefer the chosen response A over the rejected response B. In a real DPO process, this update would happen repeatedly with many pairs until the loss is very small.


## From scratch, in Python

```python
import math

# A simple representation of our model's probability for a response given a prompt.
# In real life, this is the last layer of a neural network.

def model_probability(prompt, response, theta):
    # This is a stub. In reality this would be a complex function.
    # For this example, we'll just assume a simple, deterministic mapping.
    # theta is a single number representing the model's 'state'.
    return (hash(prompt + response) % 100) / 100 * theta

def reference_probability(prompt, response):
    # The reference model is just another set of probabilities, maybe from earlier training.
    return (hash(prompt + response, ) % 100) / 100 * 0.5

# Our training data: a list of tuples (prompt, chosen_response, rejected_response)
preference_pairs = [
    ("What is the capital of France?", "Paris is the capital.", "It's London."),
]

# Hyperparameter
beta = 0.1  # This controls how much we want to deviate from the reference model.

# The model's trainable parameter
# In a real LLM, this would be the weight matrices, but we'll simplify it to a scalar.
theta = 1.0

# Training loop
for epoch in range(2):  # Do 2 passes over the data
    for prompt, chosen, rejected in preference_pairs:
        log_ratio_chosen = math.log(model_probability(prompt, chosen, theta) / reference_probability(prompt, chosen))
        log_ratio_rejected = math.log(model_probability(prompt, rejected, theta) / reference_probability(prompt, rejected))

        # The DPO loss
        logits = beta * (log_ratio_chosen - log_ratio_rejected)
        loss = -1 * math.log(1 / (1 + math.exp(-logits)))  # log sigmoid

        # Gradient of the loss with respect to theta... (simplified)
        # We just update theta to reduce loss. We'll need to compute the derivative.
        # For simplicity, we'll do a naive update that makes chosen more likely.
        # This is NOT the right way, but helps see the logic.
        delta = 0.01
        theta += delta  # increment theta (in reality, gradient descent will use the correct sign)

        print(f"Epoch {epoch}, Loss: {loss:.4f}")

```


## The same thing, with a framework

In a real framework like PyTorch or Hugging Face Transformers, you would not write this from scratch. You'd use a library like `trl` (Transformer Reinforcement Learning), which has `DPOTrainer`.

```python
from trl import DPOTrainer, DPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load a base model, often after supervised fine-tuning
model = AutoModelForCausalLM.from_pretrained("path/to/sft_model")
ref_model = AutoModelForCausalLM.from_pretrained("path/to/sft_model")  # reference is usually the same as the start point

trainer = DPOTrainer(
    model=model,
    ref_model=ref_model,
    train_dataset=your_dataset,  # formatted with 'prompt', 'chosen', 'rejected' fields
    tokenizer=AutoTokenizer.from_pretrained("model-name"),
    args=DPOConfig(beta=0.1, learning_rate=5e-6),
)

trainer.train()
```


## What the code does

The `from_scratch` code is a highly simplified illustration. In a real model, `model_probability` would be the entire neural network processing the prompt and response to output a probability. The `preference_pairs` list is a list of tuples, where each tuple holds a prompt and the chosen and rejected responses. The loop iterates over this list. Inside, we calculate the log ratios which compare the current model's likelihood to the reference model's likelihood. Then we combine them into a single 'logit' which is passed to a sigmoid function, and then we take the negative log to get the loss. The update step is a terrible simplification — in reality, you'd calculate the gradient of this loss with respect to every single weight in the neural network and then use a optimizer like Adam to update them. The framework code shows how you'd actually do this in practice. `DPOTrainer` handles all the batching, the gradient calculation, and the optimizer. You only need to provide the model, a reference to the initial model, and a dataset structured with `prompt`, `chosen`, and `rejected` columns. The `beta` hyperparameter is the same as in the formula.


## Common misconceptions

**DPO requires a reward model.** — I thought DPO still secretly trains a reward model, it just calls it something else.

This is the whole point of DPO. It is 'Direct Preference Optimization' — the loss directly tells the model how to move based on the preference, without any intermediate step that predicts a score. The formula is derived from the RLHF objective, but there is no separate model. In fact: The DPO loss function is structured so that the optimal policy (the underlying probability distribution of the model) can be derived directly from the preference data and the reference model. The training loop updates the model's weights to minimise this loss, which mathematically nudges the model toward the preferences.

**DPO is strictly worse than RLHF.** — If RLHF was developed first and is used by openAI, it must be better than DPO.

While RLHF can be very powerful, DPO often matches RLHF performance on standard alignment benchmarks. Its simplicity makes it more stable, easier to tune, and less prone to 'reward hacking' because there's no separate reward model to exploit. In fact: DPO has been shown to be competitive with or even superior to RLHF on several standard benchmarks. Its main benefit is simplicity: it needs only a reference model and preference data, not a complex RL loop, which can fail to converge.

**DPO is just supervised fine-tuning.** — It's just another version of teaching the model to say 'this' instead of 'that', isn't it?

Supervised fine-tuning (SFT) maximizes the probability of the correct answer on its own. DPO does more: it *decreases* the probability of the rejected response and *increases* the probability of the chosen response in a way that maintains the model's general capabilities. The 'reference model' in DPO prevents it from drifting too far from its original behaviour. In fact: SFT is a prerequisite, but DPO has a 'contrastive' nature: it explicitly uses pairs to make the model prefer one over the other. The reference model in DPO is crucial; it acts as a regularizer, preventing the model from collapsing its probability into a few common outputs and destroying its diversity.

**DPO can fix any model.** — I can give my bad model some preference pairs and it will become a great model.

DPO works best when the model has already gone through SFT and has a good base of knowledge. A model that produces poor, low-quality responses will be hard to steer to high quality. It's like trying to teach a poorly trained dog a complex trick; the basics have to be there. In fact: DPO is an alignment method, not a foundation. Its effectiveness hinges on the quality of the base model and, more importantly, the quality of the preference pairs. The 'Craft of Post-Training' notes this: 'Preference data quality dominates algorithm choice.' A noisy, inconsistent dataset of preferences will lead to a confused and worse model.


## In short

DPO is a powerful but simple technique for aligning LLMs. Instead of building a separate reward model, it directly adjusts the model's probabilities to make it more likely to generate 'preferred' responses and less likely to generate 'rejected' ones. It is a pivotal advancement because it simplifies the alignment pipeline while often matching RLHF quality. Master this, and you understand the cornerstone of modern model post-training.


## Where this is used

Now, where is DPO actually used in the real world of building AI? 

**Large Language Models (LLMs):** This is the primary arena for DPO. The most common use case is 'alignment' after pre-training. A company like OpenAI or a research lab at Google will first pre-train a massive model on trillions of words, then fine-tune it on high-quality instruction-response pairs (SFT). After that, they want to make it more helpful and less harmful. They collect a dataset of prompts where human annotators have ranked several responses from best to worst. DPO is then used to nudge the model's parameters so it consistently generates the responses that are more preferred. The mechanism is exactly what we described: for each pair, we calculate the loss and adjust the weights using gradient descent, but the effect across millions of pairs is a fundamental shift in the model's behaviour. It's often called a 'final polishing' step. 

**Small Language Models (on-device):** DPO is also crucial for small models running on phones and laptops. Because these models have far fewer parameters, you can't afford the computationally heavy and unstable reinforcement learning process used in RLHF. DPO is a much simpler, more stable loss function, making it the go-to choice for aligning a 1-billion-parameter model right on a GPU cluster, and the resulting model can then be quantized and deployed to a device. The mechanism is identical, but the ability to fit training on a single GPU is what makes it practical. A developer might download a 7B model from Hugging Face, apply DPO with a small preference dataset, and then deploy it via an API, whereas RLHF might take weeks on a server farm.

**Retrieval-Augmented Generation (RAG):** In RAG, a model retrieves relevant documents from a database and uses them to answer a question. DPO can be used to directly optimize how the final answer uses the retrieved context. For instance, the model might be trained to prefer answers that correctly cite the retrieved document over answers that are more generic but factually shaky. In data collection, you gather pairs of answers: one that uses the retrieved passage correctly and one that ignores it or hallucinate. DPO then aligns the model's behaviour to better use the context. The mechanism is a direct preference on what a 'good answer with context' looks like.

**Agentic Systems:** Agents are AI systems that can plan, use tools, and take actions in a loop. DPO is used to align their step-by-step actions. It's not the prompt/response pair that's the unit of preference. Instead, you collect trajectories of actions. For the same user request, you have a successful trajectory (e.g., the agent called the right API in the right order) and a failed one (e.g., it called a wrong tool). DPO can be applied to these complete trajectories, increasing the probability of a good sequence and decreasing the probability of the bad one. The mechanism is a bit different in that the 'response' is now a list of actions, but the core logic of increasing the preferred sequence and decreasing the rejected one remains the same.

**Everyday Applied Machine Learning:** Beyond LLMs, DPO has been applied to other tasks. For instance, in **recommendation systems**, instead of optimizing for a binary click/no-click, you could have a preference over two items (the user liked A over B). In **machine translation**, one translation might be considered more fluent/factual than the other. In **speech synthesis**, one generated speech clip might sound more natural to listeners than another. The mechanism is universal: any problem where you have data that says 'this output is better than that one' can be framed as DPO. You just define the 'prompt' (input, e.g., item list for recommendation) and the 'response' (the output, e.g., the recommended item or the generated speech). The developer would use an existing library like TRL and structure their dataset accordingly.


---

Read this on the web: https://basavapurushottam.com/learn-ai/direct-preference-optimization
