Build Diffusion Model from Scratch

You build the core of a denoising diffusion model in 26 NumPy and PyTorch problems across seven sections. It starts with the Gaussian math that diffusion relies on, adds the forward noising process and its schedules, builds the time-conditioned denoising network, and trains it to predict noise. It ends with the samplers that turn pure noise back into data and a full sampling loop that runs a tiny trained network.

You should know Python and NumPy, and be comfortable writing small networks in PyTorch. Basic probability helps, in particular the normal distribution and its mean and variance. The probability study plan and the PyTorch study plan are good preparation.

1. Gaussian foundations: reparameterized sampling, the KL divergence between diagonal Gaussians, and the score of a Gaussian mixture.
2. Forward process and noise schedules: linear and cosine schedules, the closed-form forward sample, and rescaling a schedule to reach zero terminal signal.
3. The denoising network: sinusoidal timestep embeddings, time-conditioned residual blocks, spatial self-attention, and a tiny U-Net.
4. Training objective: the noise prediction loss, the forward process posterior, converting between noise, data and velocity predictions, and Min-SNR loss weighting.
5. Sampling: the DDPM ancestral step, classifier-free guidance, the DDIM step, and Karras noise levels with Euler and Heun solvers.
6. Score and flow view: denoising score matching, annealed Langevin dynamics, and conditional flow matching.
7. Scaling up and capstone: the Fréchet distance behind FID, a Diffusion Transformer block with adaLN-Zero, a full training step, and a complete DDPM sampler.

By the end you can explain and implement each part of a diffusion model, from how the training data is corrupted to how a sampler reverses it step by step. Each section ends with a quiz. For other systems built from first principles, try Build Micrograd from Scratch or Language Modeling from Scratch.