Group Relative Policy Optimization

1. Several Responses to One Prompt

During training, the model receives a prompt and generates several complete responses. These responses form a group. A batch can contain many prompts, each with its own group. Four answers to one coding problem belong together; answers to a different problem have their own comparison group.

A language model chooses each next token from a probability distribution. Sampling from that distribution allows repeated attempts to follow different paths, although it can also produce duplicate answers. In reinforcement learning, the model's rule for assigning these probabilities is called its policy. GRPO trains that policy by changing the model's weights.

Each completed response receives a reward, a numerical score from a checking procedure or a learned reward model. For a simple coding exercise, a verifier might give 1 when all tests pass and 0 otherwise. More detailed rewards can use partial credit or several criteria. The score must come from a defined evaluation rule; the model generating four answers does not tell us which answers are correct.

2. Use the Group as a Baseline

Consider four rewards: 0, 1, 1, and 0. Their average is 0.5. Subtracting that average from each score gives −0.5, +0.5, +0.5, and −0.5. We now have a direction for learning: responses above the group average receive a positive signal, while those below it receive a negative signal.

The average acts as a baseline, a reference point for interpreting a reward. A score of 0.6 could be above average in one group and below average in another. Its relative signal depends on the other attempts at that same problem. Comparing within a prompt avoids directly treating a difficult problem's scores as if they were attempts at an easier problem.

In the actor-critic form of PPO, a learned value model, called the critic, predicts expected future reward and supplies the baseline. GRPO uses the sampled group's rewards instead, so it does not need to train that critic. This was the central change introduced in DeepSeekMath. Removing a separate critic saves its training resources, though generating and scoring multiple responses still has a cost.

3. Turn Reward Differences into Advantages

The original group-normalized approach also divides each reward difference by the group's standard deviation. This number measures how spread out the rewards are around their average. Dividing by it expresses each difference on a scale determined by that group, rather than leaving it in the reward function's original units.

For 0, 1, 1, and 0, the standard deviation is 0.5 when we average squared deviations over all four responses. Dividing the differences by 0.5 gives −1, +1, +1, and −1. These values are the advantages used in this example. They indicate how a response compares with its group; they are neither probabilities nor new correctness labels.

4. What If Every Reward Is the Same?

If every response receives 0, the group mean is also 0 and every reward difference is zero. If every response receives 1, the mean becomes 1 and the differences are still zero. With safe handling of the zero standard deviation, both groups produce zero advantages. Adding a small number to the denominator cannot create a distinction that the rewards did not provide.

These two groups mean different things about performance. All correct answers are a good outcome; all incorrect answers show that the sampled attempts failed. Neither group tells this relative reward objective which of its responses to favor. A group of identical scores contributes no reward-driven policy gradient, although a separate reference penalty or updates from other groups can still change the model.

Sampling more responses can reveal a useful difference, but it cannot guarantee one. Repeating the same unsuccessful approach many times still gives the learner little to compare. Useful training requires prompts the model can explore, enough variation in its attempts, and a scoring procedure that can recognize meaningful differences.

5. From Response Scores to Token Updates

A reward scores a complete answer, but the model generates that answer token by token. In outcome-supervised GRPO, the version covered here, the same response advantage is assigned to every generated token in that response. A response with advantage +1 gives all of its sampled token choices a positive learning signal. This does not establish that each word or reasoning step was individually correct.

For each sampled token, the trainer keeps the probability recorded when the response was generated. It then evaluates that same token at that same prefix with the current model. Current probability divided by recorded probability gives a ratio: a change from 20% to 30% produces 1.5, while a change from 20% to 10% produces 0.5.

GRPO uses PPO-style clipping to limit the incentive for large changes in the favored direction. With a threshold ε = 0.2, a positive-advantage token stops gaining extra credit once its ratio exceeds 1.2. A negative-advantage token stops gaining extra credit once its ratio falls below 0.8. Changes in the unfavorable direction still receive a corrective signal. These are limits in the objective, not hard bounds on token probabilities.

The shared advantage gives the tokens a common direction, but their probability ratios and gradients differ. Prompt tokens and padding are excluded from the generated-action loss. A sampled end-of-sequence token can be included; positions after the response has ended are not additional actions.

6. The Role of the Reference Model

The old policy generated the current response group. Its recorded probabilities anchor the clipping calculation for this update round. A reference policy serves a different purpose: it provides a distribution the training process should avoid drifting too far from. It is kept fixed during the policy updates, often starting from the model at the beginning of a training phase.

The original GRPO objective includes a KL divergence penalty, which measures distributional difference from this reference. It applies the penalty directly alongside the policy objective, separately from the rewards used to form group advantages. Its coefficient β controls the strength of that regularization. β and the clipping threshold ε are independent settings, and some later training recipes omit the reference penalty.

Removing the critic does not remove the need to score responses. A learned reward model can still supply those scores, or a verifier can check them. DeepSeek-R1-Zero used rule-based accuracy and format rewards. This is a choice of reward source, not a requirement that every GRPO run use binary checks or avoid a reward model.

7. One GRPO Training Round

Start with a model that can already generate responses and a batch of prompts. For every prompt, sample a group, record the sampled tokens and their old log-probabilities, and score the completed answers. Keep the prompt-group membership intact while calculating the means, standard deviations, and advantages.

Once advantages are calculated, the collected responses, also called rollouts, can be split into smaller batches for optimizer steps. These minibatches do not redefine the prompt groups used to calculate advantages. If the trainer reuses the rollouts for several steps, the recorded old probabilities still refer to the policy that generated them. The learning rate, gradient clipping, and number of update steps remain important even though the objective includes ratio clipping.

8. Group Advantages in PyTorch

The function below isolates GRPO's group comparison. Each row contains rewards for one prompt, and each column is a response to that prompt. It computes one fixed advantage per response, with an explicit population-standard-deviation convention and safe handling of tied rewards.

import math
import torch


@torch.no_grad()
def group_advantages(rewards, stabilizer=1e-8):
    # Rows are prompts; columns are responses to that same prompt.
    if rewards.ndim != 2 or rewards.shape[0] == 0 or rewards.shape[1] < 2:
        raise ValueError("Expected [number_of_prompts, group_size >= 2]")
    if not math.isfinite(stabilizer) or stabilizer <= 0:
        raise ValueError("stabilizer must be finite and positive")

    values = rewards.float()
    if not torch.isfinite(values).all():
        raise ValueError("Rewards must be finite")

    mean = values.mean(dim=1, keepdim=True)
    std = values.std(dim=1, correction=0, keepdim=True)
    if not torch.isfinite(mean).all() or not torch.isfinite(std).all():
        raise ValueError("Reward statistics exceed the numeric range")

    advantages = (values - mean) / (std + stabilizer)
    tied = (values == values[:, :1]).all(dim=1, keepdim=True)
    return torch.where(tied, torch.zeros_like(advantages), advantages)


rewards = torch.tensor([[0., 1., 1., 0.], [1., 1., 1., 1.]])
advantages = group_advantages(rewards)
# Approximately [[-1., 1., 1., -1.], [0., 0., 0., 0.]]

The first row returns approximately −1, +1, +1, and −1. The second returns four zeros. Computing statistics along the response dimension keeps those two prompts separate. If one prompt's responses are split across devices, gather or correctly aggregate that group's reward statistics before assigning its advantages.

To build a token-level loss, repeat each response's advantage across its valid generated positions. Align every action with the logits that predict it, and use a response mask to exclude prompt and padding positions. The PPO policy-loss example shows the clipped term; its token-mean reduction is a different weighting choice from the original response-mean GRPO objective above.

This function calculates targets, not a complete trainer. Sampling, reward evaluation, token probabilities, reference regularization, and optimization remain separate. The Hugging Face GRPO trainer documentation describes configurable implementations. Check the selected reward scaling, loss reduction, and sampling conventions when connecting these pieces; implementations do not all use the original defaults.

9. Why Normalization Choices Matter

Dividing by the group standard deviation changes the strength of the signal as well as its units. With one success among four binary rewards, the successful response gets an advantage of about +1.73. With three successes, each successful response gets about +0.58. The same reward of 1 therefore has a different normalized magnitude depending on the other outcomes.

Response length introduces another choice. Averaging within each response gives a two-token answer and a ten-token answer the same total averaging weight, but assigns different weights to their individual tokens. Averaging over all valid tokens gives every token the same weight and longer responses more total weight. Neither expression can be substituted for the other without changing the training objective.

10. Collect Fresh Responses and Repeat

After an update round, the changed model generates new response groups. Their scores establish new group baselines, and their recorded probabilities become the old probabilities for the next round. More updates on an existing group reuse its evidence; generating new groups gives the training process new attempts to evaluate.

The model learns to improve the supplied reward, so the checker's limitations matter. Passing a narrow set of tests may leave real bugs undetected, and satisfying a format rule does not establish that an answer is correct. Larger groups increase generation and scoring work, while a reliable reward and suitable prompts determine whether that extra work provides useful feedback.

The result is a model with updated token probabilities. During ordinary use, it can answer a prompt with a single generated response. The groups, reward comparisons, and clipped updates belong to training; GRPO does not require a group of answers or a reward evaluator for every inference request.