What is a GPT?
Introduction
Project introduction
What is a GPT?
A GPT is trained on one simple game, which is to read some text and guess what comes next. Given "To be, or not to b", it should put a high probability on "e". It plays this game at every position of millions of windows of text, and gets a little better each time it guesses wrong.
To write, it plays the same game on its own output. It guesses a next token, adds it to the text, and guesses again. GPT stands for Generative Pre-trained Transformer. Radford et al. built the first one in 2018; GPT-2 followed in 2019, and its smallest version has 124 million parameters.
What makes it a transformer?
Text is first cut into tokens and each token becomes a vector. A stack of transformer blocks then lets every position gather information from the positions before it, through attention, and refine it with a small neural network. The last layer turns each position's vector into a score for every possible next token.
One rule makes the whole thing work, which is that a position may only look backwards. This is the causal mask. Without it, the model would see the very token it is asked to guess. Later in this project you will remove the mask on purpose and watch the training loss drop to almost zero while the model learns nothing it can use.
Predict the next character
one window, every position at onceWhat you will build
01
Text to tokens
Turn Shakespeare into characters, see why real GPTs merge them into subwords, and score guesses with cross-entropy.
02
The transformer
Write causal attention, split it into heads, and stack blocks into a full GPT you can count parameter by parameter.
03
Train and write
Watch the loss fall, then generate text with temperature, top-k and top-p, from your model and a larger pretrained one.
Start with tokens
Write one merge step of byte pair encoding, the way GPT-2 built its vocabulary.