15 ML Papers Worth Implementing From Scratch, and in What Order

Which ML papers should you implement from scratch?

Implement fifteen papers in four stages, because each paper reuses code from the ones before it. Start with how networks learn through backpropagation, Glorot initialisation, Adam, dropout and batch normalisation. Next build LeNet-5, ResNet, the LSTM and Bahdanau attention. Then implement the transformer, GPT-2 and LoRA, and finish with the VAE, DDPM and FlashAttention.

From scratch means you write the paper's core idea yourself, in NumPy for papers 1 to 5 and in PyTorch with autograd after that, without calling the library module that already implements the paper. The essay on learning machine learning from scratch covers why the effort pays off.

What order should you implement the papers in?

Implement the papers in the order below, which follows code reuse more than publication date.

Treat the list as one repository that grows a file at a time. Pick a file below to see what it imports and what proves it works.

import torch
from dropout import Dropout
from resnet import residual
from bahdanau import masked_softmax

Backpropagation sits under all fourteen later files, which is why it comes first even if you want to reach transformers quickly.

How do you implement a paper without getting stuck?

Use the same routine for every paper so that each one stays small enough to finish.

1. Find the equations, the architecture figure and the hyperparameter table, which often sits in the appendix.
2. Write down every tensor's shape, then assert those shapes in code.
3. Build the smallest version with a tiny input and a fixed seed, and prove it correct with a gradient check, a closed-form value or a library comparison.
4. Overfit 10 to 100 examples, since a correct model drives the training loss close to zero.
5. Reproduce the paper's qualitative finding, because exact headline numbers depend on compute and tuning you do not have, and note where your version departs from the paper.

Most papers hide one small detail that a quick reading misses. The notes below show each one with the check that exposes it.

Which papers teach how neural networks learn?

Papers 1 to 5 teach how networks learn, and they are the machinery inside every later model. Write them in NumPy so that you see every gradient.

1. Learning Representations by Back-Propagating Errors (1986)

This paper made it practical to compute every weight's gradient by applying the chain rule backward through the layers. Build a small tanh MLP with hand-written backward passes and train it on XOR. For y = xW + b over a batch, the gradients are xᵀδ for W, δ summed over the batch for b and δWᵀ for x, as the backpropagation lesson explains. Check every gradient against a central finite difference in float64, where relative errors near 1e-7 mean it is right and errors above 1e-3 point to a bug.

2. Understanding the Difficulty of Training Deep Feedforward Neural Networks (2010)

Glorot and Bengio showed that activations and gradients shrink or explode across layers when weights start with the wrong variance. Their compromise between the forward and backward conditions is Var(W) = 2/(n_in + n_out). Initialise a 10-layer tanh MLP with small, large and Xavier weights, then plot each layer's activation spread, which stays steady only with Xavier. For ReLU networks, He et al. later derived 2/n_in.

3. Adam: A Method for Stochastic Optimization (2014)

Adam keeps running averages of the gradient and its square, then corrects their bias toward zero in early steps. The paper's defaults are α = 0.001, β1 = 0.9, β2 = 0.999 and ε = 1e-8.

m_t = β1·m_(t-1) + (1 - β1)·g_t
v_t = β2·v_(t-1) + (1 - β2)·g_t²
m̂_t = m_t / (1 - β1^t)
v̂_t = v_t / (1 - β2^t)
θ_t = θ_(t-1) - α·m̂_t / (√v̂_t + ε)

The step counter t starts at 1 and counts updates. Starting at 0 divides by zero, and counting epochs keeps the correction too large for far too many steps. After one step from zero moments, each parameter with a non-tiny gradient should move by about α.

4. Dropout: A Simple Way to Prevent Neural Networks from Overfitting (2014)

Dropout removes random units during training, which acts like averaging many thinned networks. The paper keeps each unit with probability p and scales the weights by p at test time, while modern inverted dropout divides kept activations by the keep probability during training. PyTorch's p is the drop probability, the opposite of the paper's. Over many training passes the mean output should match the input, and the Implement Dropout (Training Mode) problem drills the inverted version.

5. Batch Normalization (2015)

Batch normalisation normalises each feature over the mini-batch with x̂ = (x − μ_B) / √(σ²_B + ε), applies y = γx̂ + β, and switches to running estimates at inference. The backward pass is the hard part, because every input's gradient depends on the whole batch, so derive it on paper before your gradient check. In training mode, each output feature should have a mean near β and a standard deviation near |γ|. The batch normalization lesson covers the layer in more depth.

Which architecture papers should you build next?

Build LeNet-5, ResNet, the LSTM and Bahdanau attention next, in PyTorch with autograd, since these papers are about structure.

6. Gradient-Based Learning Applied to Document Recognition (LeNet-5, 1998)

LeNet-5 trained a convolutional network end to end with shared weights and subsampling. Write a 2D convolution as a loop, vectorise it with an unfold, and train a modern LeNet-style network with ReLU and max pooling on MNIST. Your convolution should match torch.nn.functional.conv2d on random inputs, and a small CNN typically reaches about 99% test accuracy.

7. Deep Residual Learning for Image Recognition (ResNet, 2015)

ResNet blocks learn a residual, y = F(x) + x, so gradients pass through an identity path. Build the paper's CIFAR-10 networks with 6n + 2 layers in plain and residual versions at 20 and 56 layers, using zero-padded identity shortcuts where the shape changes. Aim for the key finding, where the deeper plain net trains worse while residual nets improve with depth, which twenty to thirty epochs on one GPU usually show.

8. Long Short-Term Memory (1997)

The LSTM added a memory cell and multiplicative gates so that error can flow back over many time steps. Build the cell from its gates, then a character-level language model. The forget gate came later, from Gers, Schmidhuber and Cummins in 2000, so implement the modern cell and start the forget bias near 1. Compare one step with torch.nn.LSTMCell after copying weights in PyTorch's gate order of input, forget, cell and output.

9. Neural Machine Translation by Jointly Learning to Align and Translate (2014)

This paper introduced attention, where the decoder scores every encoder state with e_ij = v_aᵀ tanh(W_a s_(i−1) + U_a h_j) and averages them through a softmax. Build an encoder-decoder on a synthetic task such as converting written dates to ISO format, reusing your LSTM cell even though the paper used a GRU-like unit. Mask padded positions to negative infinity before the softmax, and check that each row of attention weights sums to 1.

Which transformer and language model papers should you implement?

Implement the transformer, GPT-2 and LoRA next.

10. Attention Is All You Need (2017)

The transformer builds a sequence model from attention and feed-forward layers alone, with Attention(Q, K, V) = softmax(QKᵀ / √d_k) V. Build sinusoidal positional encoding, multi-head attention, the encoder and decoder blocks and the warmup schedule, then train on a copy task. The decoder needs both a causal mask and a padding mask, and the paper places layer norm after each residual addition. With dropout off, the model should learn the copy task to near-zero loss. The Research section breaks this paper into smaller steps.

11. Language Models are Unsupervised Multitask Learners (GPT-2, 2019)

GPT-2 is a decoder-only transformer trained to predict the next token. After a scaled-down character-level run, build GPT-2 small with 12 layers, 12 heads, a 768-dimensional model, a 1,024-token context and a 50,257-token vocabulary. It moves layer norm before each sub-block, adds a final layer norm, uses the tanh approximation of GELU, ties the input and output embeddings and scales residual weights by 1/√N at initialisation. Load the released weights and compare logits with a reference, noting that the Hugging Face checkpoint stores weights transposed relative to nn.Linear. Changing only the last token must leave earlier logits unchanged. Measure your own model with perplexity, and see the Build LLM from Scratch plan for language modelling end to end.

12. LoRA: Low-Rank Adaptation of Large Language Models (2021)

LoRA freezes the pretrained weights and learns a low-rank update, h = W₀x + (α/r)BAx, with A random and B zero at the start so training begins from the base model's exact behaviour. Wrap the query and value projections of your GPT-2 and fine-tune on a small corpus. Outputs must match the base model before training, rank 8 trains about a quarter of one percent of the parameters, and merging BA into W₀ must not change the outputs.

Which generative and efficiency papers finish the list?

The VAE, DDPM and FlashAttention finish the list, with the first two generating data and the third making attention fast on real hardware.

13. Auto-Encoding Variational Bayes (VAE, 2013)

The VAE maximises a lower bound on the log-likelihood and samples z = μ + σ ⊙ ε so that gradients can pass through the sampling. Train an encoder and decoder on MNIST with a 2D latent space you can plot. The KL term has the closed form −½ Σ(1 + log σ² − μ² − σ²), which Implement KL Divergence practises, so compare it with a Monte Carlo estimate. A KL term near zero with blurry reconstructions means posterior collapse.

14. Denoising Diffusion Probabilistic Models (DDPM, 2020)

DDPM learns to reverse a process that gradually adds Gaussian noise, training a network to predict that noise with plain mean squared error.

x_t = √ᾱ_t·x_0 + √(1 - ᾱ_t)·ε,   ᾱ_t = (1 - β_1)(1 - β_2)…(1 - β_t)
loss = ‖ε - ε_θ(x_t, t)‖²

The paper uses T = 1000 steps with β rising linearly from 1e-4 to 0.02. Train an MLP on two-moons data first, then a small U-Net on MNIST that reuses your residual blocks. Most bugs live in the sampler, which adds no noise at the final step and is easy to get off by one in t. At t = T, √ᾱ_T is about 0.006, so noised data should look like pure noise.

15. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022)

FlashAttention computes exact attention in tiles that fit in on-chip memory, so the full N × N score matrix never reaches GPU main memory. Its key idea is an online softmax that rescales partial results as each block of keys arrives, which you can write first in NumPy. The FlashAttention from scratch project walks through the same tiles and memory counts step by step.

import numpy as np

def attention_row_online(q, K, V, block=64):
    d = q.shape[-1]
    m, l = -np.inf, 0.0
    o = np.zeros(V.shape[-1])
    for s in range(0, K.shape[0], block):
        scores = K[s:s + block] @ q / np.sqrt(d)
        m_new = max(m, scores.max())
        p = np.exp(scores - m_new)
        scale = np.exp(m - m_new)
        l = l * scale + p.sum()
        o = o * scale + p @ V[s:s + block]
        m = m_new
    return o / l

This version should match plain attention to floating-point precision. Then write the forward kernel in Triton or CUDA. The speedup comes from memory traffic, so learn thread blocks, shared memory and reductions first, as the CUDA practice problems guide lays out. The Triton Kernels plan includes an online softmax and a FlashAttention forward kernel, and its submissions report timings across input sizes on different GPU models.

Can you skip papers or change the order?

You can skip papers when your goal is narrow, as long as you write the reused pieces, such as batch norm or a masked softmax, inside the later paper instead.

The swap that rarely works is jumping straight to paper 10, because without backpropagation and Adam behind you, transformer bugs look like optimisation problems. Once the list is done, GANs, DQN, BERT and rotary position embeddings make good next projects.

What mistakes do people make when implementing papers?

The most common mistake is calling the library module that already implements the paper, because nn.Transformer hides every decision you are trying to learn. Other frequent mistakes are easier to miss.

• Dropout and batch norm get tested in only one of training and evaluation mode.
• Broadcasting turns a (B,) tensor minus a (B, 1) tensor into a (B, B) result without an error.
• Two settings change at once, which hides which one helped.
• Runs have no fixed seed, so a fix cannot be told apart from noise.

The guide to ML coding interview questions covers the same habits under interview pressure.

What else do people ask about implementing ML papers?

Which TensorTonic problems practise these papers?

Each of these problems isolates one piece of a paper on the list, and you can then browse the problem library for more.

• Implement Adam Optimizer Step has you write one bias-corrected Adam update from paper 3.
• Batch Normalization (Forward) has you write the forward pass from paper 5.
• Implement Causal Masking for Attention has you build the mask papers 10 and 11 depend on.
• FlashAttention Forward Kernel in Triton has you write the tiled kernel from paper 15.

Related guides

• Learning machine learning from scratch
• CUDA practice problems
• ML coding interview questions: 20 to implement
• How to prepare for an ML engineer interview