# On-Policy vs Off-Policy

Whether training data comes from the current policy's own outputs or from a different, earlier or external source.

**Rests on:** Pre-Trained Model Adaptation, Synthetic Data Generation

*LLMs and Generative AI · Post-Training Alignment*


## Why learn this

This distinction determines how much it costs to train your AI model and what kind of data it needs. When you understand it, you can see why some training methods require a constant stream of new, fresh examples while others are cheaper because they reuse old material. It is the key that unlocks the question of why some approaches demand expensive, real-time interaction with the world, while others simply study static archives.


## The intuition

Imagine you are a chef who wants to learn to cook better. You have two ways to learn. One way is to cook a dish, taste it, and if the food is too salty, you correct it next time. The lesson comes from your own cooking. The second way is to read a recipe book written by a famous chef. You try to copy the book exactly, but you never taste your own attempt; the recipe was created by someone else, in a different kitchen. The lesson comes from an external source.

In AI, we call the model's current strategy for making decisions its 'policy'. When the training data comes from the policy's own current outputs—meaning the model tests itself and learns from those specific results—we call that 'on-policy' learning. This is like the chef tasting their own food. It allows the model to adapt to its own style and immediate environment, but it requires generating new data every time.

When the training data comes from a different source—such as an earlier version of the model, a human expert, or any external archive—we call that 'off-policy' learning. This is like the chef reading a book written by someone else. The lessons come from a fixed record. It can be trained more cheaply because the data is static, but the lessons may not fit the model's current situation or style.


## An analogy

Think of a student driver learning to drive a car. On-policy learning is like taking driving lessons with an instructor in the passenger seat. The student makes a mistake, the instructor tells them immediately ('you are too close to the curb'), and the student learns from their own actions. The data (the driving mistakes and corrections) comes directly from the student's current driving performance.

Off-policy learning is like studying a driving manual written by a professional decades ago. The manual contains general rules ('keep two car lengths behind the vehicle ahead'), but it was written from someone else's experience, not the student's own. The student reads it and tries to follow the rules, but if the student has a specific problem (e.g., a tendency to hug the left side of the road), the manual may not address it, because the book was written for a general audience, not for this particular driver.

The key difference: on-policy uses the student's own driving data to improve; off-policy uses someone else's data. The analogy breaks down slightly because an AI does not have a body or senses—it only knows numbers—but the structural relationship between the source of the feedback and the learner remains the same.


## Definition

On-policy learning is training a model using data generated by its own current decision-making process, whereas off-policy learning is training it using data from a different source, such as an earlier version of itself or human experts.


## How it connects

This concept sits beside Supervised Fine-Tuning, which is a method where the model learns by copying examples provided by humans rather than generating its own. It also connects to Reinforcement Learning with Human Feedback, a process where the model improves by interacting with a system that judges its responses in real time. On-policy methods typically involve such real-time interaction, while off-policy methods often rely on static datasets.


## The mathematics

This concept does not require advanced mathematics, but it does involve a core idea from probability and statistics: the concept of a 'distribution'. In plain words, a distribution tells you which outcomes are likely. For example, when you roll a fair die, the distribution of outcomes is that each number (1 to 6) is equally likely.

For an AI's policy, the distribution is over possible responses. Given a prompt, the policy produces some answers more often than others. We can think of the policy as a machine that rolls a weighted die every time it wants to answer.

The distinction between 'on-policy' and 'off-policy' comes down to where the training data comes from. Training data is just a list of examples (a prompt and an answer) used to teach the AI.

On-policy means that the data used for training comes from the distribution of the current policy. That is, we ask the current version of the AI to generate answers, and we use those generated answers to train it further. The data matches the policy's own behavior.

Off-policy means that the data comes from a different distribution. For example, the data might be a fixed dataset collected by a human expert, or by an earlier, frozen version of the policy. The data does not match the current policy's behavior.

The key mathematical operation is not a formula but a comparison: are the samples drawn from the same distribution that we are trying to improve, or from a different one? The maths of 'probability distribution' is the branch of mathematics called 'probability'. We need it because we are talking about randomness: the policy does not always produce the same answer; it samples from a distribution. The concept itself is about comparing two distributions: the current policy's distribution and the data's distribution. If they are the same, it is on-policy; if they are different, it is off-policy.

On-policy: the training data D is sampled from the policy's own distribution π_current. Off-policy: the training data D is sampled from a fixed distribution π_data, where π_data ≠ π_current.


## A worked example

Imagine a toy AI that will answer a single prompt: 'What is the capital of France?'. We have a current policy that has learned some behaviour but is not perfect. We also have a static dataset of examples written by a human expert.

Step 1: Current policy output distribution: with 70% probability it says 'Paris', 20% 'Lyon', 10% 'Marseille'.   # This is the current policy's behaviour. It is not a single answer but a set of probabilities.
Step 2: Static dataset (from a human expert): 100% of examples say 'Paris'.   # This is the external data source. It is fixed and does not change.
Step 3: On-policy step: sample 10 responses from the current policy and use those 10 responses to train.   # The training data resembles the policy's current distribution: some 'Paris', some 'Lyon', some 'Marseille'. The model learns to make 'Paris' more likely and 'Lyon' less likely.
Step 4: Off-policy step: use the static dataset of 100 'Paris' responses to train.   # The training data is all 'Paris', so the model learns to always say 'Paris'. But if the current policy has a different distribution (e.g., says 'Lyon' often), this data may not fit.
Step 5: After training on-policy, the policy distribution changes to: 90% 'Paris', 5% 'Lyon', 5% 'Marseille'.   # The policy has adapted to its own mistakes. On-policy data is current.
Step 6: After training off-policy on the static 'Paris' data, the policy distribution becomes 95% 'Paris', 5% 'Lyon', 0% 'Marseille'.   # The policy has learned from a fixed external source. It may not reflect the policy's own situation.

Result: On-policy training uses data that matches the current policy's own distribution, so the model improves in a way that is directly relevant to its current behaviour. Off-policy training uses a fixed external distribution, which may be cheaper but can cause the model to ignore its own current behaviour patterns.


## From scratch, in Python

Let's build a tiny simulation in Python to see the difference. First, we need to understand a couple of Python building blocks. A 'list' is a collection of items in order, like a shopping list. For example: `responses = ['Paris', 'Lyon', 'Marseille']`. A 'dictionary' looks up a value by a key, like a real dictionary looks up a word to find its definition. For example: `weights = {'Paris': 0.7, 'Lyon': 0.2, 'Marseille': 0.1}`. A 'loop' repeats a block of code a certain number of times. For example, `for i in range(3): print(i)` prints 0, 1, 2. We'll also use a 'function', which is a reusable block of code that can be called with inputs. For example, `def add(a, b): return a + b`. Now, the code:

```python
# A simple policy: it has a probability distribution over answers.
import random

def current_policy():
    # Return a random answer based on the policy's probabilities.
    r = random.random()
    if r < 0.7:
        return 'Paris'
    elif r < 0.9:
        return 'Lyon'
    else:
        return 'Marseille'

# On-policy: generate 1000 responses from the current policy and use them for training.
def on_policy_train(iterations=1000):
    # Count how many responses we get.
    counts = {'Paris': 0, 'Lyon': 0, 'Marseille': 0}
    for _ in range(iterations):
        answer = current_policy()
        counts[answer] += 1
    return counts

# Off-policy: use a static dataset collected from a human expert.
# The static dataset has only 'Paris' responses.
static_data = ['Paris'] * 1000  # a list of 1000 'Paris' strings

def off_policy_train(iterations=1000):
    # Count how many responses we draw from the static data.
    counts = {'Paris': 0, 'Lyon': 0, 'Marseille': 0}
    for i in range(iterations):
        answer = static_data[i]  # always 'Paris'
        counts[answer] += 1
    return counts

# Run both and compare.
on_counts = on_policy_train()
off_counts = off_policy_train()
print('On-policy counts:', on_counts)
print('Off-policy counts:', off_counts)
```


## The same thing, with a framework

In a real framework like PyTorch, you might use a data loader to sample batches from your dataset. On-policy would require you to generate new data using the current model parameters, while off-policy would load data from a file. For example, in a reinforcement learning library, off-policy methods store transitions in a 'replay buffer' and sample from it, while on-policy methods discard old data after each update.

```python
# Pseudocode for a PyTorch-style training loop
# On-policy
for episode in range(100):
    states, actions, rewards = generate_rollout(policy_model)
    loss = compute_loss(policy_model, states, actions, rewards)
    optimizer.zero_grad(); loss.backward(); optimizer.step()

# Off-policy
replay_buffer = load_static_dataset()
for step in range(1000):
    states, actions, rewards = replay_buffer.sample(batch_size=32)
    loss = compute_loss(policy_model, states, actions, rewards)
    optimizer.zero_grad(); loss.backward(); optimizer.step()
```


## What the code does

Line by line: first we import the `random` module to generate random numbers. The function `current_policy` returns 'Paris' with probability 0.7, 'Lyon' with probability 0.2, and 'Marseille' with probability 0.1. In `on_policy_train`, we loop 1000 times, each time calling `current_policy()` to get a sample from the current policy, and we count the results. The counts will be roughly 700-200-100, matching the policy distribution. In `off_policy_train`, we use a static list of 1000 'Paris' strings, which acts as our external data source. The loop simply counts each 'Paris' response. The counts will be exactly 1000-0-0. This shows that on-policy training data reflects the current policy's distribution, while off-policy training data reflects the external source. In practice, training would adjust the policy's weights based on these samples, but this simplified version shows the key difference: the source of the data samples.


## Common misconceptions

**'Off-policy means the policy is not used at all.'** — Off-policy methods never use the current policy to make decisions.

Off-policy methods do use the current policy to act in the environment, but the training data comes from a different source. For example, in Q-learning, the agent acts with the current policy but learns from old experiences stored in a buffer. In fact: The distinction is not about whether the current policy acts, but about whether the training data comes from its own outputs.

**'On-policy is always better than off-policy.'** — On-policy learning is strictly superior because it uses current data.

On-policy learning is more sample-efficient in adapting, but it is much more expensive because it requires generating new data each step. Off-policy methods are cheaper and can reuse data, making them practical in many real-world scenarios. In fact: Each has tradeoffs. On-policy adapts better but costs more; off-policy is cheaper but may use stale data.

**'Off-policy data is always outdated and useless.'** — Static datasets are always stale and should be avoided.

Static datasets can be excellent if the task is stable. For example, SFT uses human demonstrations that are annotated and high-quality, even though they are off-policy. The issue is only if the policy's distribution has drifted far from the data distribution. In fact: Off-policy data can be valuable and is often the only data available; the key is to monitor how much it differs from the current policy.

**'On-policy and off-policy are the same if the dataset is big enough.'** — If I have a huge static dataset, it effectively becomes on-policy.

No amount of data can make a static dataset match the current policy's distribution if the policy has changed. The distribution is defined by the policy's own outputs, not by the size of the data. In fact: The data must be sampled from the current policy to be on-policy; a big static dataset is still off-policy.


## In short

On-policy learning uses the current policy's own outputs as training data, off-policy uses a different source. On-policy adapts to the current policy but costs more; off-policy is cheaper but may drift. The post-training stack in the library uses both: SFT is off-policy, while DPO and GRPO are on-policy.


## Where this is used

In large language models, on-policy vs off-policy is central to post-training. For Supervised Fine-Tuning (SFT), the model is trained on human demonstrations — that is off-policy, because the data comes from humans, not the model itself. For Direct Preference Optimization (DPO), the model samples two responses for the same prompt using its current weights, then a human or model labels which is better. That is on-policy because the data (the two responses) comes from the current policy's distribution. For GRPO, the model samples a group of responses and ranks them relative to each other; again, on-policy. In reinforcement learning from human feedback (RLHF), a reward model is trained off-policy on human preferences, and then the policy is updated on-policy by generating new responses and getting reward scores from that fixed reward model. The key tradeoff: on-policy methods are more sample-efficient in the sense that they adapt to the current policy, but they require generating new data each step, which is expensive. Off-policy methods can reuse a fixed dataset, which is much cheaper, but the data may be stale — the policy may have changed so much that the old examples no longer reflect its current behaviour. This distinction drives operational cost: a production system that does on-policy training must continuously gather new data, often from live users, while an off-policy system can train offline on historical data. In agentic systems that plan and call tools, the policy generates trajectories (a sequence of actions); on-policy training uses those trajectories to improve the plan, while off-policy training uses a dataset of successful trajectories from a different agent. In small models running on devices, on-policy training is often impossible because the device cannot afford to generate many samples; instead, they distill knowledge from a larger teacher model, which is off-policy. In retrieval-augmented generation (RAG), the generation policy is often fine-tuned on data from the retriever — that is off-policy, because the retriever's outputs may not match the current policy's style. In everyday applied work like recommendation or fraud detection, the model is often trained on historical logs of user behaviour, which is off-policy because the current model was not the one that generated those logs. The danger: the model may learn to exploit the old data in ways that do not generalise to the current situation. That is the drift problem in your notes.


---

Read this on the web: https://basavapurushottam.com/learn-ai/on-policy-vs-off-policy
