Back to Study Plans
62 Problems

Language Modeling from Scratch

62 PyTorch, NumPy, CUDA, and Triton problems covering the modern language-model stack.

15 Easy30 Medium17 Hard
17 sections
Train a Deterministic BPE VocabularyApply Ranked BPE MergesTokenization Quiz10 questions
Named-Dimension Batched Attention ScoresMixed-Precision Training Memory AccountantTransformer Training FLOP EstimatorGradient Accumulation EquivalencePyTorch and einops Quiz10 questions
RMSNorm Forward PassParameter-Matched SwiGLU BlockRotary Query and Key EmbeddingsCausal Grouped-Query AttentionArchitectures Quiz10 questions
Parallel and Recurrent Linear AttentionMamba 2 Gated State ScanGated DeltaNet State UpdateTop-k MoE Router with Load StatisticsAttention Alternatives Quiz10 questions
GPU Occupancy CalculatorShared-Memory Bank Conflict AnalyzerGlobal-Memory Coalescing CounterBlockwise Online SoftmaxGPUs and TPUs Quiz10 questions
Masked Triton GELU KernelTriton Row-Wise SoftmaxTiled Triton Row SumFused Tiled Matmul and ReLUKernels, Triton, and XLA Quiz10 questions
Simulate Distributed CollectivesCalculate Collective Effective BandwidthSynchronize Data-Parallel GradientsReconstruct a Column-Parallel MLP LayerParallelism I Quiz10 questions
Account for ZeRO Memory by StageMeasure Pipeline Bubble UtilizationChoose a Topology-Aware Parallel PlanParallelism II Quiz10 questions
Fit a Data Power Law with a Noise FloorExtrapolate Data-Mixture Scaling CurvesEstimate Critical Batch SizeRecover an IsoFLOP Compute-Optimal FrontierScaling Laws I Quiz10 questions
Decode Attention with a KV CachePlan KV-Cache CapacityPerform Exact Speculative SamplingInference Quiz10 questions
Generate a Warmup-Stable-Decay ScheduleFit Learning-Rate and Batch-Size ScalingOrthogonalize a Muon Momentum UpdateDiagnose muP Width InvariantsScaling Laws II Quiz10 questions
Compute Conditional PerplexityScore Multiple-Choice ContinuationsFit Elo Ratings from Pairwise VotesRemove Response-Length Bias from Judge ScoresEvaluation Quiz10 questions
Perform Stable Exact DeduplicationBuild Deterministic MinHash SignaturesCompute Temperature-Based Data MixturesData Processing Quiz10 questions
Build an Assistant-Only SFT Loss MaskCanonicalize Pairwise Preference DataScore a KL-Regularized Policy BatchImplement the DPO LossMid-Training and Post-Training Quiz10 questions
Normalize GRPO Group AdvantagesCompute a Clipped GRPO SurrogateCompare Sequence and Token Length NormalizationPost-Training with RLVR Quiz10 questions
Patchify a Batch of ImagesImplement Symmetric CLIP Contrastive LossImplement the SigLIP Pairwise LossAssign Multimodal RoPE CoordinatesAlignment and Multimodality Quiz10 questions
Simulate Continuous Decode BatchingFind Reusable Prefixes in a KV TrieRoute Cache-Aware Prefill RequestsMeasure Kernel Timeline UtilizationGuest Lecture with Dan Fu Quiz10 questions

What you will learn

This plan follows the modern language-model stack end to end, in 62 PyTorch, NumPy, CUDA and Triton problems across 17 sections. You go from training a tokenizer through model architecture, GPU kernels, parallel training and scaling laws, to inference, evaluation, data processing and post-training.

Prerequisites

You should be comfortable with Python, PyTorch and the basics of how a Transformer works. The PyTorch study plan and the attention mechanism are good preparation. Some sections involve GPU kernels; the Triton study plan helps there.

The sections, in order

  1. Tokenization: train a byte-pair encoding vocabulary and apply its merges.
  2. PyTorch, architectures and attention alternatives: einops-style attention scores, memory and FLOP accounting, RMSNorm, SwiGLU, rotary embeddings, grouped-query attention, linear attention, state-space scans and mixture-of-experts routing.
  3. GPUs, TPUs and kernels: occupancy, bank conflicts, memory coalescing, online softmax and Triton kernels.
  4. Parallelism: collectives, data and tensor parallelism, ZeRO memory, pipeline bubbles and choosing a parallel plan.
  5. Scaling laws: power-law fits, data mixtures, critical batch size, compute-optimal frontiers, learning-rate schedules, Muon and muP.
  6. Inference and evaluation: KV-cache decoding and capacity planning, speculative sampling, perplexity, multiple-choice scoring, Elo ratings and judge bias.
  7. Data and post-training: deduplication, MinHash, data mixtures, SFT loss masks, preference data, KL-regularized policies, DPO and GRPO.
  8. Multimodality and systems: image patches, CLIP and SigLIP losses, multimodal RoPE, and serving problems from a guest lecture on batching, prefix caching and kernel utilization.

Where to go next

For the serving side in more depth, continue with the Inference Engineering study plan. To read about the ideas behind the problems, see GPT pretraining, the KV cache and GRPO.