Build from first principles

Build a Tiny GPT from Scratch

Turn Shakespeare into tokens, build every part of a decoder-only transformer yourself, train it until the loss falls, and generate new text with temperature, top-k and top-p.

What is a GPT?See how a model that only guesses the next character can write

1Merge the Most Frequent PairText to tokens

2Next-Token Cross-Entropy over a BatchNext-token prediction

3Causal Self-Attention with a MaskCausal self-attention

4Multi-Head Causal Attention: Split and MergeMulti-head attention

5Count the Parameters of a GPTWhere the parameters live

6Temperature, Top-k and Top-p Sampling FilterSampling text

From Attention to a Tiny GPTSee how your functions fit together before training

Train and Sample a Tiny GPTBuild the model in the browser, train it, generate Shakespeare, then break the mask on purpose