Reinforcement Learning from Human Feedback

1. What Pretraining Teaches a Model

During pretraining, a language model reads sequences of tokens, the pieces into which text is divided, and learns to predict what comes next. Its weights, the numbers adjusted during training, change so that the tokens actually present in the training text become more likely. Repeating this across a large collection of text teaches language patterns, information, and ways of solving tasks.

That objective does not directly ask whether a response follows an instruction or is appropriate for a particular reader. A technical textbook explanation and a beginner-friendly explanation can both be plausible text. If the prompt asks for a simple answer, we need a way to express that the second is a better fit, even when both are grammatically correct.

This is the gap that instruction tuning and preference training address. They build on the pretrained model rather than teaching language from the beginning. The InstructGPT paper presents a widely used three-stage recipe: supervised fine-tuning, reward-model training, and reinforcement learning against that reward model.

2. Start with Examples of Good Answers

Supervised fine-tuning, usually shortened to SFT, trains the model on prompts paired with desired responses. For a request such as "Explain overfitting in one sentence to someone new to machine learning," a demonstration supplies an answer written for that audience. The model learns to make the demonstrated answer more likely in that context.

The underlying training task is still next-token prediction. A training loss measures prediction error. In a common assistant-training setup, the prompt is provided as context and this loss is calculated on the demonstrated assistant response. The model sees the earlier correct response tokens while predicting the next one, a procedure called teacher forcing. This teaches response format, instruction following, and examples of useful behavior.

A prompt can have several good answers, and writing demonstrations for every situation is expensive. Comparing existing responses offers another kind of information: which of two answers better meets the request. SFT provides a useful starting assistant, while the following stages learn from these relative judgments. SFT itself is supervised learning; the reinforcement learning stage comes later.

3. Collect Comparisons Between Responses

To collect preference data, generate multiple responses to the same prompt and ask reviewers to compare them using a stated rubric. Depending on the task, that rubric may include accuracy, relevance, clarity, instruction following, and appropriate handling of unsafe requests. Reviewers need the prompt because a good response depends on what was asked.

A simple training record contains three fields: the prompt, a chosen response, and a rejected response. Here, "rejected" means less preferred in this comparison. It may still be a correct answer, and the chosen response may still have flaws. A relative preference does not certify either response as perfect.

For the overfitting prompt, both responses in the visual below describe the same general issue. The first avoids specialist vocabulary and is likely to fit the requested audience better. That distinction is the supervision we want to preserve. Changing the label in the visual also shows why the quality of the collected judgments matters.

Real datasets contain disagreements, ambiguous prompts, and ties. A collection process should allow reviewers to flag those cases rather than forcing every pair into a confident binary choice. The simple loss in the next section assumes a strict preference; ties require an explicit treatment in the data or objective.

4. Train a Model to Predict Those Preferences

A reward model reads a prompt together with a completed response and returns a single numerical score. A common implementation uses a Transformer with a small scoring head, a learned layer that maps a sequence representation to one number. Instead of training that head to predict the next vocabulary token, we train the reward model to assign a higher score to the chosen response.

The training loss compares the two scores. If the chosen response receives the lower score, the loss is larger; if it receives a convincingly higher score, the loss is smaller. Across many labeled comparisons, the model learns patterns associated with the preferences in the dataset. This reward-model training is also supervised learning.

A raw reward score is not a probability, a factuality rating, or a percentage of human approval. In the common pairwise formulation, the difference between two scores is converted into a predicted preference probability. Adding the same constant to both scores leaves that prediction unchanged. A score of 3 from one reward model therefore cannot be treated as universally better than a score of 2 from another.

import torch
import torch.nn.functional as F


def preference_loss(chosen_scores, rejected_scores):
    return -F.logsigmoid(chosen_scores - rejected_scores).mean()


# Two trainable scores illustrate one comparison, not a full reward model.
scores = torch.nn.Parameter(torch.zeros(2))
optimizer = torch.optim.SGD([scores], lr=0.5)

for step in range(8):
    optimizer.zero_grad(set_to_none=True)
    loss = preference_loss(scores[0:1], scores[1:2])
    loss.backward()
    optimizer.step()

    with torch.no_grad():
        probability_a = torch.sigmoid(scores[0] - scores[1])
        print(step + 1, scores.tolist(), probability_a.item())

Once the reward model has learned from human comparisons, it can score new responses without asking a reviewer to judge every training sample. The Learning to Summarize from Human Feedback work uses this separation between learning a reward from comparisons and optimizing a text-generating model against it.

5. The Language Model Becomes a Policy

In reinforcement learning, a policy specifies how an agent chooses actions. For a language model, the context is the prompt plus the tokens already generated, and the next action is a token. The model's next-token probabilities are its policy. It can sample a token, add it to the context, and continue until it produces a complete response.

Generating responses for training is often called collecting rollouts. The reward model scores each completed response, providing feedback on the policy's own attempts without requiring a demonstrated target answer for every prompt. Training aims for a higher expected reward, meaning a higher average score over responses sampled from the policy, subject to constraints we will introduce next.

Sampling a token is a discrete choice. In this pipeline, we do not backpropagate through the sampled text and the reward model as though they formed one ordinary differentiable network. A policy-gradient update instead uses the model's log-probabilities for the tokens it generated, weighted by a learning signal derived from the rewards. That is how feedback on a completed response can influence future token choices.

6. Keep a Reference for the Original Behavior

Optimizing only the reward score can push the policy toward patterns that the reward model scores highly even when people would not prefer them. Classic RLHF therefore keeps a frozen reference model, commonly a copy of the SFT model from the start of reinforcement learning. It supplies a baseline distribution of responses while the trainable policy changes.

KL divergence measures how different one probability distribution is from another. Here, it measures the policy's departure from the reference. A common objective subtracts a scaled KL penalty from expected reward. The coefficient β, pronounced beta, controls the strength of this penalty: larger values make departures more costly at a fixed reward scale.

The reference is evaluated on the policy's generated tokens to obtain their probabilities; it does not need to write a competing response. Its weights stay fixed in this recipe. The penalty gives the policy a reason to retain useful starting behavior while improving the reward, although it cannot guarantee that behavior will remain correct or safe. Hugging Face's RLHF overview describes the reward and reference components together.

7. One RLHF Training Round, End to End

We now have an SFT-initialized policy, a trained reward model, and a frozen reference. A classic PPO-based trainer also uses a value model, or critic. At each response prefix, it estimates the expected remaining return: the reward accumulated from that point onward, including any penalty terms used by the trainer.

Comparing the observed return with this baseline helps estimate an advantage: how much better or worse an action turned out than expected in its context. A positive advantage encourages that token choice; a negative one discourages it. Practical trainers commonly combine estimates across several token positions to make this learning signal less noisy.

The reward model and value model answer different questions. The reward model judges a completed response using learned preferences; the value model predicts expected return from a partially generated response so training has a baseline. In a common arrangement, the response-level reward arrives at the end, while reference penalties are applied along the tokens. Return and advantage estimation carry that feedback back to earlier decisions.

8. How PPO Updates the Policy

Proximal Policy Optimization, or PPO, is one way to perform the policy update. After collecting a rollout batch, it can take several optimization steps on that batch. As the weights change, it compares each sampled token's current probability with the probability recorded when the rollout was generated.

PPO uses a clipped objective to reduce the incentive for an excessively large change in the direction encouraged by an advantage. Suppose a token has a positive advantage and its probability rises from 0.20 to 0.30. That is a ratio of 1.5. With an illustrative clipping threshold of 0.2, the clipped term stops rewarding this increase beyond a ratio of 1.2. Clipping the objective does not impose a hard bound on every probability change.

This rollout-time policy and the reference policy serve separate purposes. PPO compares against the policy that generated the current batch to control a local update. The KL penalty compares against the fixed reference to discourage drift across the training run. The saved rollout probabilities are refreshed when a new batch is collected; the reference normally stays fixed for this phase.

RLHF describes the source of the training signal, so it is not tied to PPO. Other policy-gradient methods can use a learned human-preference reward. PPO is useful to study here because it makes the policy, reward, reference, and value roles explicit in one established pipeline.

9. A Higher Reward Is Only Part of the Goal

A reward model approximates the judgments in its training data. It can learn useful preferences, but it can also learn shortcuts. If reviewers often favor longer answers, the model may reward length even when additional text adds little value. Policy training can amplify that shortcut until scores improve while answers become less useful. This is one form of reward hacking.

Research on reward-model overoptimization studies how pushing harder on a learned proxy can eventually reduce performance under a stronger evaluation signal. The reference penalty can limit movement, but it cannot make an inaccurate reward model correct. Evaluation must therefore include judgments and tasks beyond the reward score being optimized.

Preferences also depend on who provides the labels, the instructions they receive, and which prompts are represented. Reviewers may disagree for valid reasons, and a persuasive answer can be mistaken for a factual one. A useful evaluation checks held-out prompts, factual correctness where it can be established, instruction following, and potential regressions. The survey of RLHF limitations explains why collecting feedback and defining an adequate reward remain separate challenges.

10. What Happens After Training

The resulting assistant is the updated policy. In ordinary serving, it generates tokens using its trained weights without running the reward model, reference model, or training critic for every reply. A product can add separate ranking or safety systems, but those are additional serving decisions. Feedback collected from users may inform a later training run; this pipeline does not automatically update the model during each conversation.

Preference training also has alternatives to the explicit reward-model-and-PPO loop. Direct Preference Optimization, or DPO, trains a policy directly from chosen and rejected responses using a preference objective and a reference model. The DPO paper derives this approach without fitting a separate reward model or collecting fresh rollouts inside the standard offline optimization loop. It uses similar preference data through a different training procedure.

In the classic RLHF pipeline, demonstrations establish an assistant's starting behavior, comparisons define the preferences to learn, and reinforcement learning changes which responses the policy tends to produce. The quality of that change depends on the feedback and its learned approximation. Better instruction following is a training outcome to verify, rather than a guarantee supplied by the method's name.