How to Learn Reinforcement Learning from Scratch, One Stage at a Time

How do you learn reinforcement learning?

You learn reinforcement learning by writing each algorithm yourself in the order the ideas build on each other, starting with bandits and the Bellman equations and ending with PPO and the objectives used to post-train language models. Almost every method in the field is a way of estimating returns, values or advantages and then using them to improve a policy, so the early tabular stages teach the pieces that the deep methods reuse. TensorTonic has a coding problem with tests for each of these steps, and the Cracking Reinforcement Learning study plan collects 30 of them into six sections, from discounted returns to PPO, DPO and GRPO.

This guide works as a reinforcement learning roadmap in twelve stages. Each stage has a short explanation and a table that pairs every topic with a page to read and a problem to solve, so you can learn RL from scratch one tested function at a time and then train a full agent with PPO.

What is reinforcement learning, and how is it different from supervised learning?

Reinforcement learning is the part of machine learning in which an agent learns to choose actions by trying them and receiving rewards, with the goal of collecting as much reward as possible over time. At each step the agent sees a state, picks an action, and the environment returns a reward and the next state. The agent never gets told the right action, so it has to discover which actions pay off, often long after it took them.

Supervised learning is different in ways that shape everything later in this guide. A supervised model learns from a fixed dataset of inputs and correct labels, while an RL agent collects its own data, and what it sees depends on how it currently behaves. A supervised loss tells the model exactly how wrong each prediction was, while a reward only says how good an outcome was, and it may arrive many steps after the action that caused it. The agent therefore has to explore to find better actions and to assign credit for a reward to the actions that earned it.

What do you need to know before learning reinforcement learning?

You need Python, NumPy and some probability before you start learning reinforcement learning, and PyTorch before you reach the deep RL stages. The probability centres on expected values, since a value function is an expected return and every Bellman equation is an expectation written out as a sum. You do not need measure theory or convergence proofs to write working code.

• For probability, you should be comfortable with expectations, conditional probability and sampling, which the Probability and Statistics study plan covers.
• For PyTorch, you should be able to write a small network and a training loop by the time you reach DQN, and the guide to learning PyTorch and the PyTorch study plan cover both.
• For gradients, you should understand the chain rule well enough to see why a loss on log probabilities moves a policy.

The ML Math lessons below cover the background that the early stages lean on most, and you can return to them when a stage needs it.

In what order should you learn reinforcement learning?

You should learn reinforcement learning from the simplest decision problem to the most general one, which means bandits first, then MDPs and the Bellman equations, then methods that learn values from experience, and only then neural networks and policy gradients. Each stage adds one difficulty to the one before it, such as delayed rewards, an unknown model or a state space too large for a table. The Cracking Reinforcement Learning plan teaches MDPs before bandits, and either order works because bandits stand mostly on their own.

Stage 1: Why start with bandits and exploration?

Bandits are the best starting point because they isolate the exploration problem, with no states and no delayed rewards. A multi-armed bandit offers several actions, each with an unknown reward distribution, and the agent must balance trying arms it knows little about against pulling the arm that looks best so far. Epsilon-greedy explores at random with a fixed probability, UCB1 adds a bonus that shrinks as an arm is pulled more often, and Thompson sampling draws a guess for each arm from its posterior and pulls the arm with the best guess.

The same exploration ideas return in every later stage. Epsilon-greedy is the usual exploration rule in Q-learning and DQN, softmax exploration is what a policy network does when it samples from its output distribution, and contextual bandits such as LinUCB are used directly in recommendation and advertising systems.

Stage 2: What are MDPs and the Bellman equations?

A Markov decision process is the standard model of a problem in which actions affect future states, made of states, actions, transition probabilities, rewards and a discount factor gamma. The quantity the agent maximises is the discounted return, the sum of future rewards with each one multiplied by gamma once more for every step it lies in the future. The Bellman expectation equation says that the value of a state under a policy equals the expected immediate reward plus gamma times the expected value of the next state.

The Bellman optimality equation replaces the average over the policy with a maximum over actions, and its solution gives the best possible value of each state. Every algorithm in the next ten stages either solves one of these two equations or approximates it from samples, so it is worth computing a few backups by hand on a tiny MDP before you go on.

Stage 3: How does dynamic programming solve an MDP?

Dynamic programming solves an MDP by applying Bellman backups to every state again and again until the values stop changing. Value iteration applies the optimality backup on each sweep and reads off the greedy policy at the end, while policy iteration alternates between evaluating the current policy and making it greedy with respect to the new values. Both converge to the optimal policy on a finite MDP with gamma below one.

The weakness of dynamic programming is that it needs the full transition probabilities and reward function, and it visits every state on every sweep. Most real problems give you neither, which is why the rest of reinforcement learning is about learning the same values from sampled experience.

Stage 4: How do Monte Carlo and TD methods learn from experience?

Monte Carlo and temporal-difference methods learn values from episodes the agent actually experiences, without a model of the environment. Monte Carlo evaluation waits until an episode ends and moves each visited state towards the return that followed it, which is an average of samples in exactly the sense of the Monte Carlo lesson. TD(0) updates after a single step, moving the estimate towards the reward plus gamma times the current estimate of the next state, and the gap between that target and the old estimate is called the TD error.

Using the current estimate inside the target is called bootstrapping, and it is the most important idea in this stage. It lets TD learn before an episode ends and in tasks that never end, at the cost of some bias while the estimates are still wrong, and it appears again in Q-learning, DQN, actor-critic methods and GAE.

Stage 5: How do Q-learning and SARSA learn to act?

Q-learning and SARSA learn to act by estimating the value of each action in each state, so the agent can improve its behaviour by picking the action with the highest estimate. SARSA moves Q(s, a) towards the reward plus the discounted value of the action the agent actually takes next, so it learns the value of the policy it is following, exploration included. Q-learning uses the largest next action value instead, so it learns the value of the greedy policy whatever the agent does while exploring.

Expected SARSA averages the next action value over the policy instead of using one sampled action, which lowers the variance of the target. A tabular Q-learning agent on a small grid world is the first point in this roadmap where you can watch an agent learn a task, so it is worth training one before moving on.

Stage 6: Why does deep Q-learning need a target network and a replay buffer?

Deep Q-learning needs a target network and a replay buffer because training a neural network on its own bootstrapped targets, with highly correlated samples, is unstable. Function approximation replaces the Q table with a network so that similar states share what they learn, but each gradient step then shifts the targets as well. DQN computes its targets with a copy of the network that is updated only occasionally, and it trains on random minibatches drawn from a buffer of past transitions.

Each later improvement fixes one specific weakness. Double DQN reduces the upward bias that comes from taking a maximum over noisy estimates, dueling DQN learns the value of a state separately from the advantage of each action, and prioritized replay samples transitions with large TD errors more often and corrects for that with importance weights.

Stage 7: How do policy gradients and REINFORCE work?

Policy gradient methods train a network that outputs a distribution over actions, and they push up the log probability of each action in proportion to how good the outcome after it was. REINFORCE uses the full return after each action as that weight, which follows from the log-derivative trick, where the gradient of a probability equals the probability times the gradient of its log. The trick turns the gradient of an expected return into an average you can estimate from sampled episodes, even though the environment itself is not differentiable.

Policy gradients handle continuous and very large action spaces naturally, since the network outputs the parameters of a distribution and never has to take a maximum over actions. The price is high variance, and subtracting a baseline such as a learned state value is the first and most effective fix, because it lowers the variance without biasing the gradient.

Stage 8: What do actor-critic methods and GAE add?

Actor-critic methods add a learned value function, the critic, that estimates how much better each action was than average, and the policy, the actor, uses that estimate as its weight in place of the raw return. The estimate is called the advantage, and generalized advantage estimation builds it as a discounted sum of TD errors controlled by a parameter lambda. A lambda of zero gives the one-step TD error, with low variance and some bias, and a lambda of one gives the full return minus the value, with no bias from the critic and high variance.

The A2C loss combines three terms, a policy loss weighted by the advantage, a squared error loss for the value function and an entropy bonus that keeps the policy from becoming deterministic too early. Treating the advantage as a constant in the policy term is a detail that matters, since gradients from the policy loss should not flow into the critic.

Stage 9: Why is PPO the default policy gradient method?

PPO is the default policy gradient method because it lets you reuse each batch of experience for several epochs of updates without the policy moving too far from the one that collected the data. It computes the ratio of the new probability of each action to the old one, multiplies it by the advantage, and clips the ratio to a small range around one, often 0.8 to 1.2. Taking the minimum of the clipped and unclipped terms removes any reward for large changes while keeping the full penalty when a change makes things worse.

Two numbers tell you whether PPO is healthy, the approximate KL divergence between the old and new policies and the clip fraction, the share of samples whose ratio fell outside the clipping range. Both rising steadily usually means each update is too large. PPO is also the algorithm that RLHF uses to fine-tune language models, so it connects this stage directly to stage 12.

Stage 10: How do you build an environment and design a reward?

You build an environment as an object with a reset method that starts an episode and a step method that takes an action and returns the next observation, the reward and whether the episode has ended. Modern RL libraries split the ending into two flags, terminated for when the task itself is over and truncated for when a time limit cut it short. The difference matters because a learning target should stop adding future value only when the episode is terminated, and treating a time-out as a real ending teaches the agent wrong values.

Reward design is where most practical RL goes wrong, because an agent optimises the reward exactly as written. A bonus for moving closer to a goal, for example, can be farmed by stepping towards it and away again forever. Potential-based shaping, which adds gamma times a score of the new state minus the score of the old state, gives the same kind of guidance without changing which policy is optimal. The Build an RL Environment project has you build a grid world, watch a Q-learning agent game a closer-to-the-goal bonus, replace it with one that cannot be farmed, find a time-limit bug and compare each reward across ten random seeds.

Stage 11: How does AlphaGo combine search and self-play?

AlphaGo Zero combines a neural network that predicts good moves and the likely winner with Monte Carlo tree search, and it trains the network on games it plays against itself. Each search simulation walks down the tree using the PUCT rule, which favours moves with high values, high network priors and few visits so far, and the value at the new leaf is backed up along the path with its sign flipped at each level. After many simulations, the visit counts at the root form a stronger policy than the network alone.

Self-play turns that search into training data. Dirichlet noise is added to the priors at the root so that the search tries unusual moves, the normalised visit counts become the target for the policy head, and the final game result becomes the target for the value head. The Build AlphaGo from Scratch study plan builds this loop in 18 problems across five sections, starting with the rules of Go.

Stage 12: How is reinforcement learning used to train language models?

Reinforcement learning trains language models by treating the model as a policy, each generated token as an action and a score for the finished response as the reward. RLHF trains a reward model on pairs of responses that people compared, using a Bradley-Terry loss that pushes the score of the preferred response above the other, and then optimizes the language model with PPO plus a KL penalty that keeps it close to a frozen reference model. The KL penalty matters because without it the model drifts towards text that scores well with the reward model and reads badly to people.

DPO skips the reward model and the sampling during training, and it learns directly from preference pairs with a loss on the log probability ratios between the model and the reference. GRPO keeps sampling but drops the critic, scoring each response against the mean reward of a group of responses to the same prompt, and RLOO does the same with a baseline built from the other samples in the group. Each of these is a short loss on top of the policy gradient ideas from stages 7 to 9.

Can you learn reinforcement learning without a GPU?

Yes, you can learn reinforcement learning without a GPU, because the tabular methods, bandits and small control tasks all run quickly on a CPU. Even DQN and PPO on a task like CartPole use networks with a few thousand parameters, and the slow part of training is usually stepping the environment, which runs on the CPU anyway. A GPU only starts to matter for agents that learn from images, large models and RL on language models.

The coding problems linked in this guide run in the browser and are checked against tests, so there is nothing to install. Our guide on practising ML without a GPU covers where to find compute when you want to train something larger.

How do you put the pieces together into a trained agent?

You put the pieces together by training a PPO agent on a small control task, checking each part on its own before you trust the learning curve. The Train an Agent with PPO project follows this path in PyTorch on a CartPole written in plain Python, training REINFORCE and PPO on the same data, comparing them across 5 seeds, and then removing the clipping to watch training collapse and find the warning in the approximate KL and clip fraction.

1. Run a random agent first and record how long its episodes last, so you know what no learning looks like.
2. Build a policy network and a value network, and check that the log probabilities and entropy match a hand calculation on a small batch.
3. Collect a rollout and compute discounted returns that reset correctly at every episode boundary.
4. Train REINFORCE with a baseline and confirm that episodes get longer before adding anything else.
5. Add GAE and the clipped objective, and log the approximate KL and clip fraction on every update.
6. Train PPO with a few epochs of minibatch updates per batch and compare it with REINFORCE on several seeds.

What should you build after your first PPO agent?

After your first PPO agent, the most useful next steps are building an environment of your own, combining search with learning, and applying policy optimization to a language model. Each one takes a stage of this roadmap further, and the table pairs each direction with something to read and a project or plan to work through.

The Align a Language Model with DPO project turns preference pairs into a loss, trains a rude help desk model into a polite one, and measures how far the model drifts when beta is too small. If you are preparing for interviews, the reinforcement learning interview questions guide has model answers for each topic in this roadmap, and the LLM interview questions guide goes further into RLHF, DPO and GRPO.

What mistakes do people make when learning reinforcement learning?

• Jumping straight to deep RL on a hard environment makes every bug look like a tuning problem, so get tabular Q-learning working on a small grid first.
• Forgetting the done mask lets a terminal state borrow value from the next episode, and it is one of the most common real bugs in DQN and GAE code.
• Treating a time limit as a real ending gives the agent wrong values near the limit, so keep terminated and truncated separate.
• Judging a method on one random seed overstates what you learned, since RL results vary widely between seeds, so compare methods on several.
• Watching only the reward hides problems such as a collapsing entropy or a growing KL, so log those alongside the score.
• Letting gradients from the policy loss flow into the critic, or forgetting to stop gradients through the target network, quietly breaks training.
• Trusting a shaped reward without watching the trained agent lets reward hacking go unnoticed, because the score can rise while the behaviour gets worse.

Practice this

Then work through the full Cracking Reinforcement Learning study plan, or train an agent end to end in the Train an Agent with PPO project.

Related guides

• Reinforcement learning interview questions with answers
• LLM interview questions with answers
• How to learn PyTorch, from tensors to a training loop
• How to learn diffusion models from scratch, one stage at a time
• 15 ML papers worth implementing from scratch, and in what order
• Maths for machine learning