From Attention to a Tiny GPT

You have written the core of a GPT, which is a byte pair encoding merge, the next-token loss, causal attention, multi-head attention, the sampling filter, and the parameter count of a whole model.

In the notebook these pieces become a character-level GPT with about 108,000 parameters, trained on eight Shakespeare plays right in your browser. Seven of its cells ask you to write one small piece, using the same ideas you just practised, and check it before you move on. The rest is written for you and explained.

What you will find out

In under 20 seconds of training the loss falls from 4.17, a blind guess among 65 characters, to about 2.1, below a model that only looks at the previous character. Its text has the shape of a play, with real short words mixed with made-up ones. A larger GPT trained for longer, loaded into your own class, writes text that reads like Shakespeare at a glance.

Then you remove the causal mask. The training loss collapses to almost zero, far better than the real model, yet the generated text is garbage. You will measure the loss at every position and find exactly where the model cheats.

Predict the next character

one window, every position at once

To␣be,␣or␣not␣to␣be

position 0 reads itself and everything before itand must guess "o"later positions are masked

Tokens65 characters to ids

Embedtoken row + position row

Blocksmasked attention + MLP

Sampletemperature, top-k, top-p

How the notebook is organized

Build and train

Tokens, the loss, attention, heads and blocks become a GPT, and you watch its loss fall.

Generate

Write your sampling filter and compare greedy decoding, high temperature and nucleus sampling.

Break the mask

Train without the causal mask, watch the loss collapse, and find where the model cheats.

Train and sample a tiny GPT

Run the whole build in the browser, about 45 minutes.

Open the notebook project