GPT Pretraining

The Training Example Hidden in Text

GPT pretraining looks complicated because a serious run involves enormous datasets, thousands of accelerators, and weeks of coordination. The learning problem at the center is much smaller. Take a piece of text, hide the next token, and ask the model to predict it from the tokens on its left.

The target row is just the input row shifted one place to the left. Seven input positions become seven supervised predictions. No one had to annotate a subject, write a question, or choose a label. The text provides both the context and the answer.

What Pretraining Means

GPT stands for Generative Pre-trained Transformer. Generative means the model defines a probability distribution over continuations. Transformer names the neural architecture. Pre-trained means these weights are learned before the model is adapted to a narrower job such as following instructions, answering in a chat format, or writing code in a house style.

The original GPT work used a two-stage recipe: learn a general language model from unlabeled text, then fine-tune it on labeled tasks. GPT-2 pushed harder on the first stage and showed that a larger next-token model could perform some tasks from text alone. GPT-3 scaled the same broad autoregressive setup further and studied zero-shot, one-shot, and few-shot behavior without changing weights at evaluation time.

1. The Data Pipeline

Pretraining starts before the first GPU sees a batch. A corpus is assembled from documents, code, books, reference material, or other licensed and permitted sources. The exact mixture is part of the model design. It decides which patterns the model sees often, which languages receive capacity, and which errors will be repeated enough to learn.

Deduplication has two jobs. It stops frequently copied pages from consuming a disproportionate share of updates, and it reduces overlap between training data and evaluation sets. GPT-2 already treated n-gram overlap as a measurement problem. At larger scales, weak deduplication can make a benchmark look like reasoning when part of the answer was present in the training corpus.

2. Text Becomes Tokens

A transformer does not receive words. It receives integer IDs from a fixed vocabulary. The tokenizer converts a string into a sequence of tokens, and an embedding table converts each ID into a learned vector.

The split above is illustrative because the exact pieces depend on the tokenizer. GPT-2 used byte-level BPE. It begins from bytes, then learns frequent adjacent merges. Common strings can become one token, while unusual strings fall back to smaller pieces without requiring an unknown-token escape hatch for ordinary byte sequences.

Tokenization is not cosmetic. It sets the sequence length seen by the model, the size of the input and output matrices, and the unit used by cross-entropy. Perplexities from two different tokenizers are therefore not directly comparable.

3. Inputs and Shifted Labels

Tokenized documents are separated by an end-of-document token and packed into fixed-length windows. If a training sample contains $T+1$ tokens, the first $T$ are the model input and the last $T$ are the target.

This is often called teacher forcing. At position $t$, the model receives the real prefix from the dataset, not tokens sampled from its own earlier predictions. That makes training parallel and stable, but it also creates a difference from generation, where the model must condition on whatever it generated a moment ago.

4. The Causal Mask

The full target sequence is in GPU memory during training. Without a mask, self-attention at position $t$ could read position $t+1$, which contains the answer it is supposed to predict. Training loss would collapse for the wrong reason.

After softmax, entries receiving $-\infty$ have probability zero. The attention matrix is lower triangular: a token can use itself and its left context, never its right context.

Interactive: Causal Mask

Select a query row to see which keys it can read and which next token it must predict. Notice that later rows receive more context, but none can see its target.

5. Every Position Predicts

GPT generates sequentially, but it does not pretrain sequentially. During generation, token $t+1$ does not exist until token $t$ has been sampled. During pretraining, the complete ground-truth sequence already exists. The causal mask preserves the legal information boundary, so all positions can run through the transformer together.

This distinction is easy to miss and it explains why training hardware is built around large matrix multiplications, while serving has a separate set of problems around KV cache memory, batching, and per-token latency.

6. Through the GPT Stack

Each token ID indexes a row in the embedding matrix. Positional information is added or applied, then the sequence passes through a stack of decoder blocks. A modern block usually contains causal self-attention, an MLP, normalization, residual connections, and dropout during training.

The model has no separate grammar table, fact store, or code parser. The same weights are reused at every sequence position. Whatever structure helps next-token prediction must be represented inside those shared matrices and the activations they produce.

7. Logits and Softmax

The final hidden vector at each position has width $d$. The language-model head projects it to $V$ numbers, one for every vocabulary token. These raw scores are logits.

Softmax turns the logits into a probability distribution. Increasing one logit increases that token's probability and decreases the share left for the rest. The output matrix is often tied to the input embedding matrix, which reuses the same learned token geometry on both sides of the network.

8. Cross-Entropy Loss

For one position, the loss is the negative natural logarithm of the probability assigned to the correct next token:

If the correct token receives probability 0.9, its loss is about 0.105. At probability 0.1, the loss is about 2.303. At probability 0.001, the loss is about 6.908. The logarithm punishes confident mistakes sharply and gives the optimizer a useful signal even when the correct token is not the top prediction.

Padding tokens and sometimes cross-document positions are removed from the average with a loss mask. What remains is one scalar, even though it summarizes thousands or millions of token predictions across the effective batch.

Interactive: Token Loss

Move the probability assigned to the true next token. The other candidate bars share the remaining probability mass. Watch how slowly loss falls near certainty and how quickly it rises near zero.

9. Backpropagation and Update

Cross-entropy tells us how wrong the batch was. Backpropagation tells us which parameters contributed to that error. Automatic differentiation applies the chain rule from the scalar loss, through the vocabulary head, every transformer block, and back to the embeddings.

AdamW does more than subtract the raw gradient. It keeps moving averages of first and second moments, rescales updates, applies weight decay to selected parameters, and uses a learning rate schedule. Large GPT runs commonly warm the learning rate up, then decay it over training. Gradient clipping, mixed precision, and loss scaling are engineering safeguards around the same update.

tokens = next_batch()
inputs = tokens[:, :-1]
targets = tokens[:, 1:]

logits = model(inputs)
loss = cross_entropy(logits, targets)

optimizer.zero_grad(set_to_none=True)
loss.backward()
clip_grad_norm_(model.parameters(), max_norm)
optimizer.step()
scheduler.step()

Real systems add distributed collectives, gradient accumulation, checkpointing, fused kernels, and failure recovery. They do not change the conceptual loop shown above.

Interactive: Training Step

Adjust the batch and sequence dimensions, then step through the five phases. A batch of $B$ sequences with length $T$ supplies $B\times T$ next-token targets before masking.

10. Validation and Perplexity

Training loss measures fit on batches used for updates. Validation loss runs the same objective on held-out text without changing the weights. If training loss keeps falling while validation loss stalls or rises, the model is memorizing the training distribution faster than it is improving on unseen text.

Perplexity can be read as an effective number of equally plausible choices, but only as an intuition. A vocabulary contains tens of thousands of uneven choices, and tokenization changes the units. Compare perplexity only when the dataset, tokenization, context treatment, and loss masking are aligned.

A low held-out language-model loss is useful, but it is not a complete evaluation. Teams also measure downstream capabilities, memorization, bias, factuality, safety, multilingual behavior, code behavior, and performance across document sources.

11. Model, Data, and Compute

Once the training loop works, the main allocation question is how to spend a fixed compute budget. More parameters increase capacity. More training tokens provide more evidence. More compute lets the optimizer apply more updates. Starve any one of the three and the other two hit diminishing returns.

The early scaling-law work found smooth power-law relationships between loss and model size, dataset size, and training compute. Chinchilla later showed that many large models were undertrained for their parameter count. With the same compute budget as the 280B-parameter Gopher model, a 70B-parameter model trained on 1.4 trillion tokens performed better across the reported evaluation suite.

12. Training Across GPUs

A large GPT model may not fit on one accelerator, and one accelerator may take years to process the planned tokens. Distributed training splits the work along several axes.

Gradient accumulation adds another lever. Several microbatches run forward and backward before one optimizer update, which creates a larger effective batch without storing every activation at once. Activation checkpointing saves memory by discarding selected intermediate activations and recomputing them during backward.

The difficulty is no longer the loss formula. It is keeping expensive hardware busy while communication, memory, numerical stability, checkpoint writes, and failed workers all compete with useful matrix multiplication.

13. What the Objective Learns

Next-token prediction sounds narrow, but a good prediction often requires structure. To complete a sentence, the model benefits from syntax. To continue a function, it benefits from tracking variables and indentation. To finish a paragraph about a historical event, it benefits from associations between names, dates, places, and typical explanations.

This does not mean the objective verifies truth, discovers causal models, or stores knowledge like rows in a database. It rewards predictive usefulness on the training distribution. A fluent falsehood can lower loss if similar falsehoods are common in the data. An obscure fact may never receive enough signal to be represented reliably.

14. Pretraining Is Not Chat Training

A base GPT model continues text. If the prompt looks like an article, it continues an article. If it looks like source code, it continues code. If it looks like a conversation, it may continue any participant because pretraining alone does not define which role it should play.

These stages may still use a token-level loss somewhere in the implementation, but their data and objective are different. Saying that a chat assistant is “just next-token prediction” skips the post-training that determines how the base model is presented and controlled.

15. What Can Go Wrong

The Whole Pipeline

That is GPT pretraining. The scale changes, the architecture details evolve, and the systems engineering becomes harder. The central contract stays fixed: given the tokens on the left, assign more probability to the token that came next.