# GRPO

Group relative policy optimisation, which scores a group of sampled responses against each other rather than against a learned value function.

**Rests on:** Gradient Descent, Reward Modeling

*LLMs and Generative AI · Post-Training Alignment*


## Why learn this

GRPO is the engine behind the recent wave of reasoning models — the AI systems that write out their step-by-step thinking before giving a final answer. It allows an AI to learn how to think better by comparing its own draft attempts against each other, without needing a separate, complex 'critic' system to grade them. Understanding GRPO helps you see why earlier methods were so resource-heavy and why this new approach is both cheaper and more effective for tasks like math or logic, where there is often no single perfect answer, only better or worse paths.


## The intuition

Imagine you are a cricket coach picking players for a team. You don't have a perfect formula for who is good. But you can see the players who performed in the trial match. You put them in groups: say, five bowlers at a time. You watch them bowl. You rank them against each other: this one was better than that one. You don't need an absolute score like 7.3 out of 10 — you just need to know who did better within the group. Then you tell the players: 'you who bowled well, keep doing that; you who bowled poorly, change something.' Over many trials, the team improves.

GRPO is exactly this, applied to a language model. The model sees a question. It samples several different answers (the group). It scores each answer relative to the others — not against an absolute standard, but simply 'is this answer better than the others in this group?' Then it adjusts its weights so that answers ranked higher become more likely, and answers ranked lower become less likely. There is no separate 'critic' model telling it an absolute score; the group itself provides the signal.

Why does this matter? Older methods (like PPO in RLHF) trained a separate critic — a value network — to estimate how good every state was. That doubles memory and compute. GRPO throws the critic away. The group's own relative ranking is the signal. This is much simpler and cheaper, and it turned out to be especially powerful for training models to reason step-by-step, because reasoning tasks often have no single 'perfect' answer — just better and worse paths.


## An analogy

Think of a cooking competition. The judges are not given a scoring rubric out of 10. Instead, five contestants cook the same dish, and the judges simply say: 'this one is better than that one, and that one is the best of the five.' No one says a dish is 'a 7.3' — only which is better than which. That is relative scoring. In traditional RLHF (PPO), the judges would first be trained to give absolute scores ('this dish is a 6.5 out of 10'), and then a separate 'judge model' would be maintained to score every new dish. That is heavy. GRPO says: forget the absolute scores. Just taste the five dishes in each round and rank them.

The analogy breaks down somewhere: in a cooking contest, the judges are human and have consistent taste. In GRPO, the 'judge' is often the same model that is being trained — or a simple rule. For math problems, the judge might just check if the final answer is correct. For code, it might check if the code runs. For open-ended questions, the model itself might be used as a judge (self-rewarding). So the signal can be noisy or even biased by the model's own preferences. But the key insight holds: you don't need an absolute value function; relative comparisons within a group carry enough information to learn.


## Definition

Group Relative Policy Optimisation (GRPO) is a reinforcement-learning method that improves a model's responses by sampling a group of responses to the same prompt, scoring each response relative to the others in the group (rather than against a learned value function), and updating the model's weights to make better-ranked responses more likely.


## How it connects

You already have notes on post-training alignment, which is the overall stack. GRPO sits in the preference-optimisation stage — after Supervised Fine-Tuning (SFT) and alongside methods like DPO and ORPO. The library notes say: 'the stack is layered: SFT first, then preference optimisation, then RL where warranted.' GRPO is a form of RL where the reward signal comes from relative group comparison.

Your notes mention that DPO and ORPO skip the explicit reward model. GRPO also avoids a learned value network — but it still uses a reward signal per response (e.g., correctness or human preference), which it then normalises within the group. So GRPO is closer to traditional RL than DPO is, but it borrows the relative-comparison spirit of preference methods.

You also have notes on On-Policy vs Off-Policy. GRPO is on-policy: the responses it scores are sampled fresh from the current model at each training step, not from an old version. This is a key property — it means the model always learns from its own current distribution, which reduces the risk of the 'off-policy drift' your notes mention.


## The mathematics

This concept uses **probability and statistics** for scoring, and a bit of **calculus** for the weight update — but at the beginner level, we focus on the arithmetic of relative scoring.

First, let's understand what a 'score' is. Each response in the group gets a raw score from some judge: for a math problem, it might be 1 if the answer is correct, 0 if not. Or it could be a continuous score from a human evaluator, say 8.2 out of 10. But raw scores are not directly comparable across groups: one group might have all high raw scores, another all low. To compare across groups, we need to standardise the scores — we convert each score to a 'relative' score that says 'this is how many standard deviations above or below the group average it is.'

This is called **standardisation** (or z-scoring). It is a basic statistics operation. Let's explain it from scratch.

Suppose you have five scores: 2, 4, 6, 8, 10. The average is 6. The standard deviation measures how spread out the numbers are. Roughly, how far a typical score is from the average. For these numbers, the standard deviation is about 3.16. Now, the standardised score for each number is: (score − average) ÷ standard deviation. For 2, that's (2−6)÷3.16 = −1.26. For 10, that's (10−6)÷3.16 = 1.26. Negative means below average, positive means above.

In GRPO, after standardising, the model updates its weights using a formula that increases the probability of responses with positive relative scores and decreases the probability for negative ones. The exact calculus formula (the policy gradient) is not needed at the beginner level, but the intuition is: the model adjusts itself in the direction that makes better-than-average responses more likely.

The key point: relative ranking within a group is robust to overall bias. If the judge is consistently generous (giving everyone high raw scores), the relative scores still identify the best of the group. That is why GRPO can work with a simple reward signal without a learned value function.

For a group of outputs g, with each output x having reward r(x) (e.g., correctness), the standardised score is:

A_g = (r(x) − μ_g) / σ_g

where μ_g is the group mean reward and σ_g is the group standard deviation. This normalised advantage is then used in a policy-gradient update:

∇θ J ≈ (1/(G·T)) Σ_g Σ_t A_g · ∇θ log πθ(x_t | x_{<t})

(Symbol glossary: r = reward, μ = mean, σ = standard deviation, x = a token in the response, πθ = the model with weights θ, log πθ = log-probability the model assigns to a token, ∇θ = the gradient (how to adjust weights), G = group size, T = response length.)


## A worked example

```
We have a prompt: 'What is 3×4?' The model samples 4 responses, each with a raw reward score (1 for correct, 0 for wrong). Responses:
A: '12' → reward 1 (correct)
B: '11' → reward 0 (wrong)
C: '12, I think' → reward 1 (correct)
D: '13' → reward 0 (wrong)
We will standardise these scores.

Step 1: Scores: [1, 0, 1, 0]   # These are the raw rewards for the four responses.
Step 2: Average μ = (1+0+1+0)/4 = 0.5   # The mean of the four scores.
Step 3: Variance = ((1−0.5)²+(0−0.5)²+(1−0.5)²+(0−0.5)²)/4 = 0.25   # Average squared deviation from the mean.
Step 4: Standard deviation σ = √0.25 = 0.5   # Spread of the scores.
Step 5: Standardise A: (1−0.5)/0.5 = 1.0   # A is one standard deviation above the mean.
Step 6: Standardise B: (0−0.5)/0.5 = −1.0   # B is one standard deviation below.
Step 7: Standardise C: (1−0.5)/0.5 = 1.0   # Same as A.
Step 8: Standardise D: (0−0.5)/0.5 = −1.0   # Same as B.

Result: The standardised (relative) scores are [1.0, −1.0, 1.0, −1.0]. Positive means above average, negative below. The model will be updated to make responses like A and C more likely, and B and D less likely.
```


## From scratch, in Python

```python
import random
import math

# Simulate the model as a simple distribution over 5 possible answers
def sample_response(prompt, probabilities):
    # Return an index based on the probabilities
    r = random.random()
    cumulative = 0.0
    for i, p in enumerate(probabilities):
        cumulative += p
        if r < cumulative:
            return i
    return len(probabilities) - 1

def reward_for_answer(answer_index, correct_answer_index):
    return 1.0 if answer_index == correct_answer_index else 0.0

def group_relative_advantage(rewards):
    mean = sum(rewards) / len(rewards)
    variance = sum((r - mean)**2 for r in rewards) / len(rewards)
    std = math.sqrt(variance)
    # Avoid division by zero
    if std == 0:
        return [0.0 for r in rewards]
    return [(r - mean) / std for r in rewards]

def update_probabilities(probabilities, sampled_indices, advantages, learning_rate=0.1):
    # Increase probabilities for positive advantage, decrease for negative
    new_probs = probabilities[:]
    for idx, adv in zip(sampled_indices, advantages):
        # For simplicity, adjust the probability directly (not true gradient)
        new_probs[idx] += learning_rate * adv
    # Ensure non-negative and renormalise
    new_probs = [max(p, 0.0001) for p in new_probs]
    total = sum(new_probs)
    return [p / total for p in new_probs]

# Example: correct answer is index 2 (out of 5)
correct = 2
probabilities = [0.2, 0.2, 0.2, 0.2, 0.2]  # uniform
print("Initial probabilities:", probabilities)

# Training loop (a few steps)
for step in range(5):
    # Sample a group of 4 responses
    group = []
    for _ in range(4):
        group.append(sample_response("What is 3x4?", probabilities))
    print("Group:", group)
    rewards = [reward_for_answer(a, correct) for a in group]
    print("Rewards:", rewards)
    advantages = group_relative_advantage(rewards)
    print("Advantages:", advantages)
    probabilities = update_probabilities(probabilities, group, advantages)
    print("Updated probabilities:", probabilities)

```


## The same thing, with a framework

In practice, GRPO is implemented in PyTorch with the model generating responses, computing rewards, and then doing a policy-gradient update. The following sketch shows the key steps in a simplified form:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("my-model")
tokenizer = AutoTokenizer.from_pretrained("my-model")
optimizer = torch.optim.Adam(model.parameters(), lr=1e-6)

prompt = "What is 3x4?"
inputs = tokenizer(prompt, return_tensors="pt")

# Generate a group of responses (greedy or sampled)
with torch.no_grad():
    outputs = [model.generate(**inputs, max_new_tokens=10, do_sample=True) for _ in range(4)]

# Compute rewards: e.g. check if the answer contains "12"
rewards = []
for out in outputs:
    text = tokenizer.decode(out[0], skip_special_tokens=True)
    rewards.append(1.0 if "12" in text else 0.0)

# Standardise rewards within the group
mean = torch.tensor(rewards).mean()
std = torch.tensor(rewards).std()
advantages = (torch.tensor(rewards) - mean) / (std + 1e-8)

# Compute the policy gradient loss (simplified: negative log probability weighted by advantage)
loss = 0
for output, adv in zip(outputs, advantages):
    logits = model(input_ids=output, attention_mask=torch.ones_like(output), labels=output).logits
    # This is a placeholder: real implementation computes the log-prob of the response tokens
    # and multiplies by the advantage.
    loss = loss - adv * logits.mean()  # placeholder
loss.backward()
optimizer.step()
```


## What the code does

The from-scratch code simulates a tiny model. Let's walk through:

- `sample_response` uses a random number to pick an answer according to the current probabilities. It simulates the model's sampling.
- `reward_for_answer` gives 1 if the answer is correct, else 0.
- `group_relative_advantage` computes the mean and standard deviation of the rewards, then standardises each. This is the 'relative' part — a correct answer in a group where all are correct gets a 0 advantage, not 1.
- `update_probabilities` adjusts each answer's probability by adding the advantage times a learning rate, then renormalises. This is a crude simulation of the policy gradient step.
- The training loop runs several 'steps', each sampling a group, computing advantages, and updating probabilities. Over time, the probability of the correct answer increases.

The framework version is the real thing: it uses PyTorch to generate responses with the model, compute rewards, standardise them, and then compute a loss that weights the log-probability of the generated tokens by the advantage. The `loss = ...` line is a placeholder — the actual implementation uses a proper policy-gradient formula (e.g., `-advantage * log_prob` summed over tokens).


## Common misconceptions

**GRPO needs a separate reward model** — GRPO requires a trained reward model to score each response, just like RLHF.

GRPO removes the value network (critic), but the reward signal itself can be a simple rule (e.g., correct answer check) or a reward model, but it does not learn a value function. The 'relative' part is the group standardisation, not the reward model. Actually, GRPO often uses a trained reward model too, but it does not need a value network. The key is the critic is gone, not necessarily the reward model. In fact: GRPO replaces the learned value network (critic) with the group mean reward, but the reward itself can come from a simple function or a reward model. The 'relative' part is just standardisation, not a learned value.

**Relative scores mean absolute scores don't matter** — As long as we standardise, the raw reward values don't matter — only the ranking within the group.

Standardisation uses the mean and standard deviation, so the spread still matters. If all rewards are 0.9 or 1.0 (all nearly perfect), the relative scores will be tiny, and the learning signal will be weak. The absolute level matters for variance. Also, if rewards are noisy, relative scores become meaningless. In fact: Relative scores depend on the distribution of rewards in the group. If all rewards are similar, the standard deviation is small, but the advantages will be small or zero, giving weak signal. So the reward function needs enough variation across responses.

**GRPO is the same as DPO** — GRPO is just another way of doing direct preference optimisation, like DPO.

DPO directly optimises on preference pairs (chosen vs rejected) without any reward model or reinforcement learning. GRPO is a reinforcement learning method that uses a policy gradient update on the model's own sampled responses. It does not need paired preference data; it can use a scalar reward (e.g., correctness). They are distinct families, though both avoid a learned value function. In fact: GRPO is an RL method: it samples responses, scores them, and updates the policy with a gradient. DPO is a supervised method that directly optimises the policy on paired preferences. They are different approaches to alignment.

**Group size doesn't matter much** — Any group size works; bigger is always better but you can just use a small group to save compute.

Group size determines the noise of the advantage estimates. With a group of 2, the standard deviation is coarse, and the advantage signal is noisy. With a group of 16 or 32, the estimates are more stable, but the compute cost increases linearly. The paper's ablations show that group size around 16 is a good tradeoff. Too small leads to unstable training. In fact: Group size is a key hyperparameter. Larger groups give more stable advantage estimates but cost more compute. Typical choices are 16–64. Too small a group can make training unstable.


## In short

GRPO is a reinforcement learning method that trains a model by sampling a group of responses to the same prompt, scoring each response (e.g., correctness), standardising those scores within the group to get a relative advantage, and updating the model to make above-average responses more likely. It removes the need for a separate value network, cutting memory and compute. It is the engine behind reasoning models like DeepSeek-R1, and it works wherever you can define a reward and sample multiple outputs.


## Where this is used

GRPO is most prominently used in training **reasoning-focused LLMs** — models that produce a chain of thought before answering. The most famous example is DeepSeek-R1, which used GRPO to teach the model to reason without any supervised reasoning data. In practice, here is how GRPO is used across the areas:

- **Large Language Models**: For LLMs, GRPO is used in the post-training phase after SFT. The model is given a set of prompts (often math or coding problems). It samples multiple responses, each containing a 'thinking' part and a final answer. A reward function checks the final answer for correctness (e.g., against a known solution) or uses a separate reward model. The group relative advantage is computed, and the model is updated to increase the likelihood of responses with positive advantage. This is done over millions of steps, with a learning rate schedule. As the notes say, the learning rate schedule can waste the entire run if set wrong — too high and the model diverges; too low and it takes forever.

- **Small Models on Devices**: For small models (e.g., on phones), GRPO is used to distill reasoning capability from larger models. A small model samples responses, and a larger teacher model scores them. The small model's weights are updated with GRPO to mimic the teacher's preferences. This lets a 1B model learn to reason almost as well as a 70B model on specific tasks, while fitting on a phone.

- **Retrieval-Augmented Generation (RAG)**: In RAG systems, GRPO can optimise the 'relevance' of generated answers given retrieved documents. A prompt includes retrieved passages, and the model generates an answer. The reward might be a QA accuracy score or a human preference. By sampling multiple answers and comparing them, the model learns to use the retrieved context better. GRPO's relative nature helps because the retrieved passages may vary in quality across prompts — relative scoring makes the model robust to that variability.

- **Agentic Systems**: In agents that plan and call tools, GRPO is used to train the agent to choose actions that lead to successful task completion. For a task like 'book a flight', the agent produces a sequence of tool calls. Each full trajectory is sampled several times. The reward is whether the task completed successfully (maybe with a penalty for extra steps). GRPO compares the group of trajectories and increases the probability of the successful ones. This is a natural fit because trajectories are highly variable — relative comparison handles the variance. Developers write custom reward functions (e.g., a scoring script) and run GRPO in a training loop, often with a GPU cluster.

- **Everyday Applied Machine Learning**: Beyond language, GRPO can be applied to any sequential decision problem where you can sample multiple solutions. For example, in code generation, you sample several programs and run them on unit tests; correct ones get reward 1. In fraud detection, you might sample different classification thresholds (though that's a bit contrived) — more realistic is to use GRPO to tune a model's decision policy over time. In forecasting, you could sample multiple prediction paths. The key is that GRPO works wherever you can define a reward and sample multiple outcomes.

**What a developer actually writes**: For GRPO, the developer typically sets up a training loop using a library like `trl` (Transformers Reinforcement Learning) or a custom PyTorch loop. They write a reward function (e.g., a Python function that checks correctness), a sampling loop (using `model.generate`), and a policy-gradient update. They also set hyperparameters: group size G (often 16 or 32), learning rate, and the number of steps. If done badly — e.g., group size too small, rewards not standardised properly, or reward function giving noisy signal — the model can collapse to always producing a safe but wrong answer, or it might overfit to the reward and become 'reward hacky' (as your notes on Reward Hacking warn).

GRPO's success in reasoning models means it is becoming the default choice for training models that need to 'think' before answering, and it's increasingly integrated into open-source alignment stacks.


---

Read this on the web: https://basavapurushottam.com/learn-ai/grpo
