How to Build an LLM from Scratch, One Component at a Time

Name: TensorTonic
Author: TensorTonic

Build a small GPT-style language model yourself in six stages, from a tokenizer to attention, training and text generation, with a tested coding problem for each part.

By TensorTonic's founderLast updated November 11, 202614 min read

How do you build an LLM from scratch?

You build an LLM from scratch by writing six parts in order and testing each one before you join them. Those parts are a tokenizer with embeddings, causal self-attention, the transformer block, a cross-entropy loss with its gradients, a training loop, and a sampler that generates text. TensorTonic has a coding problem with tests for each of these parts, and this guide links to them stage by stage.

The model you end up with is a small GPT, which is a decoder-only transformer trained to predict the next token. It uses the same architecture as the large models people use every day and differs from them in size and training data. Building the small version teaches you how the large ones work, because every component is the same.

The six stages of building an LLM from scratch and what each one produces
Stage	What you write	What works at the end
1. Tokens and embeddings	A tokenizer that turns text into integer IDs, an embedding table, and position information	A batch of token IDs becomes a tensor of vectors with the right shape
2. Attention	Scaled dot-product attention, the causal mask and multi-head attention	Each position mixes in information from earlier positions only
3. The transformer block	Layer normalization, the feed-forward network, residual connections and the stacked blocks	A full forward pass from token IDs to next-token scores
4. Loss and gradients	Cross-entropy loss on the next token and a backward pass you can check	A loss value and a gradient for every weight
5. Training	Batching, AdamW, a learning rate schedule and gradient clipping	A loss that falls on held-out text
6. Generation and evaluation	Temperature, top-k and top-p sampling, perplexity and the KV cache	New text from the trained model and a number that measures it

What do you need to know before building an LLM?

You need comfortable Python, basic NumPy, matrix multiplication and the chain rule before you start building an LLM. You do not need to have trained a neural network before, since the loss and gradient stage teaches backpropagation from the beginning. PyTorch becomes useful when you train the full model, and you can learn it at that point.

For Python, you should be able to write functions, loops and classes without looking things up, and this checklist of the Python you need for ML lets you test yourself.
For NumPy, you should understand array shapes, broadcasting and indexing, which the NumPy study plan covers.
For linear algebra, you should be comfortable with dot products and matrix multiplication, and the linear algebra guide teaches both through code.
For calculus, you should know what a derivative is and how the chain rule combines derivatives.

What are the steps to build an LLM from scratch?

The steps follow the path of data through the model. Text becomes tokens and vectors, attention mixes information between positions, the transformer block processes it, the loss measures the error, training reduces that error, and generation turns the trained model back into text. Each stage below has a short explanation and a table that pairs every topic with a page to read and a problem to solve.

Stage 1: How do you turn text into tokens and embeddings?

A model works on numbers, so text first becomes a sequence of integer IDs and then a sequence of vectors. A byte-pair-encoding tokenizer learns its vocabulary by repeatedly merging the most frequent pair of adjacent symbols, and an embedding table maps each token ID to a vector the model can learn. Attention has no built-in sense of order, so you also add position information, either with sinusoidal encodings or with rotary embeddings.

Tokens and embeddings topics with a page to read and a problem to solve
Topic	Read	Practice
Splitting text into tokens	Tokenization in the Transformer	Word Count Dictionary
Training a BPE tokenizer		Train a BPE Tokenizer From Scratch
Encoding text with BPE merges		Apply Ranked BPE Merges
Padding sequences into a batch		Pad Sequences
Token embeddings	Embeddings in the Transformer
Sinusoidal positions	Transformer Positional Encoding	Sinusoidal Positional Encoding
Rotary positions	RoPE: Rotary Position Embedding Explained	RoPE: Rotary Query and Key Embeddings

A character-level tokenizer is a reasonable shortcut for a first model, because the vocabulary is tiny and you can train sooner. Writing BPE is still worth the time, since every production model uses a subword tokenizer.

Stage 2: How do you implement attention?

Attention lets each token look at other tokens and pull in the information it needs. Each position produces a query, a key and a value, the scores are dot products between queries and keys divided by the square root of the key dimension, and a softmax turns those scores into weights over the values. A language model adds a causal mask so that a position can only attend to earlier positions, which keeps next-token prediction honest.

Attention topics with a page to read and a problem to solve
Topic	Read	Practice
Queries, keys and values	Attention in LLMs: Queries, Keys and Values Explained	Named-Dimension Batched Attention Scores
Stable softmax		Numerically Stable Softmax
Scaled dot-product attention	Scaled dot-product attention	Scaled Dot-Product Attention
Causal masking	GPT Pretraining: How Next-Token Prediction Works	Causal Masking for Attention
Multi-head attention	Multi-head attention in the Transformer	Multi-Head Attention
Grouped-query attention		Causal Grouped-Query Attention

Multi-head attention runs several attention operations in parallel on smaller slices of the vector and concatenates the results. Most of the difficulty is in reshaping tensors between the batch, head, position and feature dimensions, so write the shape of every tensor as a note beside it while you work.

Stage 3: How do you build the transformer block?

A transformer block wraps attention with normalization, a feed-forward network and residual connections, and a GPT is a stack of these blocks followed by a final linear layer over the vocabulary. The residual connections add each sublayer's output back to its input, which keeps gradients flowing through a deep stack. Once you can write one block and count its parameters, the full model is the same block repeated.

Transformer block topics with a page to read and a problem to solve
Topic	Read	Practice
Linear layers and initialization		Linear Layer Forward Pass, Xavier Initialization
Layer normalization	Layer Normalization in the Transformer	Layer Normalization, RMSNorm Forward Pass
Activation		GELU Activation
Feed-forward network	Feed-forward layer in the Transformer	Evaluate a Multi-Layer Perceptron, Parameter-Matched SwiGLU Block
Dropout		Dropout (Training Mode)
The full block	The Transformer encoder block	Transformer Decoder Block
Parameter and FLOP counts		Transformer Training FLOP Estimator

Modern models replace some of the original parts. RMSNorm often takes the place of layer normalization and SwiGLU often takes the place of the GELU feed-forward network, and the table includes a problem for each so you can compare them.

Stage 4: How do you compute the loss and its gradients?

The model learns by comparing its predicted distribution over the next token with the token that actually came next. Cross-entropy loss measures that gap, and backpropagation sends the error back through every layer to produce a gradient for every weight. The Build Micrograd from Scratch study plan has 25 problems that build a small autograd engine, so you can see exactly how a backward pass is assembled.

Loss and gradient topics with a page to read and a problem to solve
Topic	Read	Practice
Next-token prediction	GPT Pretraining: How Next-Token Prediction Works	Bigram Probabilities (Add-1 Smoothing)
Cross-entropy loss	Cross-Entropy Loss: Formula and BCE vs MSE	Cross-Entropy Loss
Computational graphs	The Chain Rule in Backpropagation	Build a Scalar Expression Graph, Topologically Sort a Computation DAG
Reverse-mode autodiff	Backpropagation and Gradient Descent by Hand	Run Reverse-Mode Autodiff
Gradient checking		Finite Difference Derivative, Gradient-Check an Autodiff Graph
Softmax and cross-entropy gradient		Softmax + Cross-Entropy Gradient

Gradient checking is the habit to take from this stage. Comparing an analytic gradient with a finite-difference estimate catches mistakes that would otherwise show up much later as a model that trains badly for no visible reason.

Stage 5: How do you train the model?

Training is a loop that samples a batch of token sequences, runs the forward pass, computes the loss, runs the backward pass and updates the weights. Language models are usually trained with AdamW, a learning rate that warms up and then decays, and gradient clipping to keep updates stable. Our guide on which optimizer to use explains why AdamW is the usual choice for transformers.

Training topics with a page to read and a problem to solve
Topic	Read	Practice
Mini-batches	SGD, Mini-Batch and Learning Rate Schedules	Batch Shuffling & Mini-Batch Generator
A full training step		Apply One Full SGD Training Step, Train a Tiny Micrograd MLP
Adam and AdamW	Adam vs RMSprop vs AdaGrad Explained	Adam Optimizer Step, AdamW (Decoupled Weight Decay)
Warmup and decay		Warmup + Linear Decay LR Schedule, Cosine Annealing LR Scheduler
Gradient clipping		Gradient Clipping (Global Norm)
Gradient accumulation	Gradient Accumulation in LLM Training	Gradient Accumulation Equivalence
Mixed precision and memory	Mixed Precision Training: FP16 vs BF16	Mixed-Precision Training Memory Accountant

The first check on any training loop is whether the model can overfit a single small batch. If the loss on one batch does not fall towards zero, there is a bug in the model or the loop, and the guide on a training loss that is not decreasing lists the causes in the order to check them.

Stage 6: How do you generate text and evaluate the model?

A trained model outputs a score for every token in the vocabulary, and the decoding rule decides which token is chosen. Temperature rescales the scores before the softmax, top-k keeps only the k most likely tokens, and top-p keeps the smallest set of tokens whose probabilities add up to p. Perplexity on held-out text is the standard measure of how well the model predicts, and a KV cache makes generation faster by reusing the keys and values of tokens the model has already processed.

Generation and evaluation topics with a page to read and a problem to solve
Topic	Read	Practice
Temperature sampling		Text Generation Sampling
Top-k and top-p sampling		Temperature, Top-k, and Top-p Sampling
Perplexity	Perplexity in LLMs: Formula and Worked Example	Perplexity Computation, Compute Conditional Perplexity
Scoring answers by likelihood		Score Multiple-Choice Continuations
KV cache	KV Cache in LLMs: How It Makes Inference Fast	Decode Attention with a KV Cache
Speculative decoding	Speculative Decoding in LLMs Explained	Perform Exact Speculative Sampling

Can you build an LLM without a GPU?

Yes, you can build an LLM without a GPU, because every component can be written and tested on small arrays that run on a laptop. A GPU matters only when you train the full model, and a small character-level model keeps that training short enough for modest hardware. The model will be far weaker than a commercial one, and that is fine when the goal is to understand how it works.

The component problems linked in this guide run in the browser and are checked against tests, so there is nothing to install. Our guide on practising ML without a GPU covers where to find free compute when you want to train something larger.

How do you put the pieces together into a model that trains?

You put the pieces together by wiring them in the order the data flows, checking shapes at every step, and training on a small text file first. The Build a Tiny GPT from Scratch project does this in a PyTorch notebook, where you build a character-level GPT that covers tokens, next-token prediction, causal self-attention, the full transformer, training, and sampling with temperature, top-k and top-p.

Encode the text into token IDs and split it into a training part and a held-out part.
Build batches where the target sequence is the input sequence shifted by one position.
Run one forward pass and confirm the output has one score per vocabulary token at every position.
Check that the loss before any training is close to the natural log of the vocabulary size, which is what a uniform guess gives.
Overfit one small batch to confirm the model and the loop can learn at all.
Train on the full data, watch the held-out loss, and sample text every so often to see it improve.

What should you build after a tiny GPT?

After a tiny GPT, the useful next steps are scaling the model, fine-tuning a pretrained one, aligning it with preferences, and serving it efficiently. These are the stages a production model goes through after pretraining, and each has an explanation in the LLM Internals lessons and a project where you build it.

What to build after a tiny GPT, with a lesson and a project or plan for each direction
Direction	Read	Build
Scale the same model up	FlashAttention	Build LLM from Scratch study plan
Fine-tune a pretrained model	LoRA, QLoRA	Fine-Tune an LLM
Align a model with preferences	RLHF, DPO	DPO from scratch project
Serve a model quickly	KV cache, PagedAttention	Build an LLM Inference Server From Scratch
Shrink a model	Quantization	Quantize a Language Model to 4 Bits
Ground a model in documents	RAG	RAG from scratch project

The Build LLM from Scratch study plan is the longest of these. It has 62 PyTorch, NumPy, CUDA and Triton problems across 17 sections, and it runs from training a tokenizer through model architecture, GPU kernels, parallel training and scaling laws to inference, evaluation and post-training. If serving interests you most, the guide to inference engineering gives a build order for that path.

Which resources teach you to build an LLM from scratch?

TensorTonic teaches the build as tested coding problems, with a problem for each component, a notebook project that trains a tiny GPT, and a 62-problem study plan that continues into systems and post-training. Several well-known courses, books and videos also teach how to build an LLM from scratch, and they pair well with problems you solve yourself.

Andrej Karpathy publishes a video series called Neural Networks: Zero to Hero, which ends with building a GPT.
Sebastian Raschka wrote the book Build a Large Language Model (From Scratch), published by Manning.
Stanford teaches a course called CS336: Language Modeling from Scratch.

Whichever one you follow, write each component yourself and test it before you move on. Reading or watching an implementation is much easier than producing one, and only the second shows you what you have understood.

What mistakes do people make when building an LLM from scratch?

Writing the whole model before testing any part makes bugs very hard to locate, so test each component on a small input first.
Leaving out the causal mask, or applying it after the softmax, lets the model see the token it is meant to predict, which gives a very low training loss and useless generated text.
Shifting the targets by the wrong amount trains the model to copy its input, so check one input and target pair by eye.
Skipping gradient checks leaves errors in the backward pass that never raise an exception.
Training with a fixed high learning rate and no clipping often makes the loss diverge early, which warmup and gradient clipping prevent.
Judging the model only by its training loss hides overfitting, so measure loss or perplexity on text the model has not seen.
Starting with a large model and a large dataset slows down every experiment, and a small model on a small text file shows the same bugs much faster.

Practice this

Scaled Dot-Product Attention is the core operation of the model, and every later attention variant is built on it.
Causal Masking for Attention is the change that turns attention into something a language model can train with.
Transformer Decoder Block joins attention, normalization, the feed-forward network and the residual connections into the unit a GPT repeats.

Then train the whole model in the Build a Tiny GPT from Scratch project, or continue with the Build LLM from Scratch study plan.

Can you build an LLM from scratch in Python?

Yes, you can build a small LLM from scratch in Python. NumPy is enough to write and test each component on small arrays, including the tokenizer, attention, the transformer block, the loss and the optimizer. Most people then switch to PyTorch to train the full model, because PyTorch computes the gradients and runs the same code on a GPU.

Is building an LLM from scratch the same as fine-tuning one?

Building an LLM from scratch and fine-tuning one are different tasks. Building from scratch means writing the tokenizer, the model and the training loop yourself and training from random weights. Fine-tuning starts from a model that someone else has already trained and adapts it to a task, often by training small adapter weights with a method such as LoRA.

Should you use NumPy or PyTorch to build an LLM from scratch?

Use NumPy for the individual components and PyTorch for the full model. Writing attention or layer normalization in NumPy forces you to handle every shape and every formula yourself, which is where the understanding comes from. PyTorch is the better choice once the pieces are joined, since automatic differentiation and GPU support make training practical.

What is the hardest part of building an LLM from scratch?

The hardest part for most people is the backward pass, followed by getting the first training run to converge. Attention and the transformer block are mostly careful shape handling, while gradients are easy to get subtly wrong without any error message. Building a small autograd engine and checking its gradients against finite differences removes most of that difficulty.

Does building an LLM from scratch help in ML interviews?

Building an LLM from scratch helps in interviews that include LLM coding questions. Interviewers for LLM roles often ask candidates to implement scaled dot-product attention, a causal mask, multi-head attention, layer normalization or a sampling method. Each of these is one component of the build, so someone who has written them all has already practised those questions.