What is PPO?
Introduction
Project introduction
What is PPO?
In reinforcement learning an agent learns by trying actions and seeing what reward follows. A policy is the agent's rule for choosing actions; here it is a small neural network that turns what the agent observes into a probability for each action. Policy gradient methods improve that network directly. They make the actions that led to high reward a little more likely, and the others a little less.
The simplest version, REINFORCE, dates from 1992. It works, but it is slow, because each batch of experience gives a very noisy direction and it can only take one small step before the experience is out of date. Proximal Policy Optimization (PPO), published by OpenAI in 2017, fixes both, and it is still one of the most widely used methods, from robot control to fine-tuning language models.
The idea in two parts
First, judge each action by its advantage, which is how much better it went than a second network, the critic, expected. Generalized advantage estimation (GAE) builds advantages from the critic's one-step surprises and lets you choose between steady and unbiased with one number, lambda.
Second, reuse each batch for many small steps without letting the policy wander off. PPO compares every action's new probability with the one it had when the data was collected and stops rewarding changes beyond a small band, usually 20 percent. That clip is the "proximal" in the name.
Proximal policy optimization
CartPole, 1024 steps per iterationEpisode length per iteration (200 is perfect)
Same data for both. PPO reuses every rollout for 40 small, clipped steps.
What you will build
01
Returns and advantages
Discounted returns across episode boundaries, then GAE with a critic.
02
The clipped objective
The probability ratio, the clipped loss, and the complete PPO minibatch loss.
03
Health checks
Approximate KL and clip fraction, the numbers that warn you before a run collapses.
Start with returns
Add up discounted rewards in a rollout where episodes end and restart.