From Advantages to PPO
Introduction
Project introduction
From Advantages to PPO
You have written every piece PPO needs. These are discounted returns, GAE advantages, the clipped surrogate loss, the approximate KL and clip fraction, and the complete minibatch loss.
In the notebook they train an agent on CartPole, a pole balanced on a cart that can be pushed left or right, written in plain Python so it runs in seconds. First you train REINFORCE, then measure why it is slow, then train PPO on exactly the same amount of experience.
What you will find out
You will measure how much the gradients of separate batches disagree, how much a critic baseline helps, and how lambda trades a steady estimate against a correct one. PPO then reaches a near-perfect score in about 7 iterations where REINFORCE needs about 32.
Then you remove the clipping. Training gets faster at first and then collapses, in some seeds to episodes of about 9 steps, no better than always pushing the same way. You will read the approximate KL and clip fraction to find out why, and see that they warned from the very first update.
Proximal policy optimization
CartPole, 1024 steps per iterationEpisode length per iteration (200 is perfect)
Same data for both. PPO reuses every rollout for 40 small, clipped steps.
How the notebook is organized
01
Policy gradients
A policy network, rollouts, returns and REINFORCE, then why its gradient is noisy.
02
PPO
GAE and lambda, the clipped objective, health checks, and epochs of minibatches.
03
Break it and judge it
Remove the clipping, diagnose the collapse, and compare learning curves across 5 seeds.
Train PPO on CartPole
Run the whole investigation in the browser, about 45 minutes.