Study Plans Labs Projects Interview Pricing Feedback

Implement ML Papers From Scratch

Each track breaks one landmark paper into small steps. You implement every step in Python in the browser, and tests check your code before you move on. If you are new, start with the Transformer, or read which ML papers to implement from scratch to choose a track.

Transformers and LLMs

Convolutional networks

Sequence models and embeddings

Generative models

GPT-2

BPE Training

Byte Pair Encoding training algorithm

BPE Encode/Decode

BPE encoding and decoding

Token + Position Embedding

Combined token and learned positional embedding

Scaled Dot-Product Attention

Core attention computation

Causal Masked Attention

Attention with causal mask for autoregressive decoding

Multi-Head Attention

Multi-head causal self-attention

GELU Activation

Gaussian Error Linear Unit activation function

Layer Normalization

Pre-norm layer normalization

Feed-Forward Network

Position-wise FFN with GELU

Decoder Block

Full pre-norm transformer decoder block

Residual Weight Scaling

Initialization scaling for deep residual networks

Full Forward Pass

Complete GPT-2 forward pass

Greedy Decoding

Autoregressive greedy generation

Top-k Sampling

Temperature scaling and top-k sampling

Loading architecture visualization...