Back to Study Plans

62 Problems

Language Modeling from Scratch

62 PyTorch, NumPy, CUDA, and Triton problems covering the modern language-model stack.

15 Easy30 Medium17 Hard

17 sections

Train a Deterministic BPE Vocabulary Apply Ranked BPE Merges Tokenization Quiz10 questions

Named-Dimension Batched Attention Scores Mixed-Precision Training Memory Accountant Transformer Training FLOP Estimator Gradient Accumulation Equivalence PyTorch and einops Quiz10 questions

RMSNorm Forward Pass Parameter-Matched SwiGLU Block Rotary Query and Key Embeddings Causal Grouped-Query Attention Architectures Quiz10 questions

Parallel and Recurrent Linear Attention Mamba 2 Gated State Scan Gated DeltaNet State Update Top-k MoE Router with Load Statistics Attention Alternatives Quiz10 questions

GPU Occupancy Calculator Shared-Memory Bank Conflict Analyzer Global-Memory Coalescing Counter Blockwise Online Softmax GPUs and TPUs Quiz10 questions

Masked Triton GELU Kernel Triton Row-Wise Softmax Tiled Triton Row Sum Fused Tiled Matmul and ReLU Kernels, Triton, and XLA Quiz10 questions

Simulate Distributed Collectives Calculate Collective Effective Bandwidth Synchronize Data-Parallel Gradients Reconstruct a Column-Parallel MLP Layer Parallelism I Quiz10 questions

Account for ZeRO Memory by Stage Measure Pipeline Bubble Utilization Choose a Topology-Aware Parallel Plan Parallelism II Quiz10 questions

Fit a Data Power Law with a Noise Floor Extrapolate Data-Mixture Scaling Curves Estimate Critical Batch Size Recover an IsoFLOP Compute-Optimal Frontier Scaling Laws I Quiz10 questions

Decode Attention with a KV Cache Plan KV-Cache Capacity Perform Exact Speculative Sampling Inference Quiz10 questions

Generate a Warmup-Stable-Decay Schedule Fit Learning-Rate and Batch-Size Scaling Orthogonalize a Muon Momentum Update Diagnose muP Width Invariants Scaling Laws II Quiz10 questions

Compute Conditional Perplexity Score Multiple-Choice Continuations Fit Elo Ratings from Pairwise Votes Remove Response-Length Bias from Judge Scores Evaluation Quiz10 questions

Perform Stable Exact Deduplication Build Deterministic MinHash Signatures Compute Temperature-Based Data Mixtures Data Processing Quiz10 questions

Build an Assistant-Only SFT Loss Mask Canonicalize Pairwise Preference Data Score a KL-Regularized Policy Batch Implement the DPO Loss Mid-Training and Post-Training Quiz10 questions

Normalize GRPO Group Advantages Compute a Clipped GRPO Surrogate Compare Sequence and Token Length Normalization Post-Training with RLVR Quiz10 questions

Patchify a Batch of Images Implement Symmetric CLIP Contrastive Loss Implement the SigLIP Pairwise Loss Assign Multimodal RoPE Coordinates Alignment and Multimodality Quiz10 questions

Simulate Continuous Decode Batching Find Reusable Prefixes in a KV Trie Route Cache-Aware Prefill Requests Measure Kernel Timeline Utilization Guest Lecture with Dan Fu Quiz10 questions

Summary

Total62

Easy15

Medium30

Hard17

Progress0%

Quizzes0/17