How do I learn to implement ML papers from scratch?

Start by choosing a paper that matches your interest: Transformers for NLP, ResNet for computer vision, or GANs for generative models. Break the paper into components and implement each as a coding problem. Test your implementation against known results. Build progressively: start with RNN, then LSTM, then Transformers.

Which ML paper should I implement first?

For computer vision beginners: VGG (simple) → ResNet (residual connections). For NLP: RNN → LSTM → Transformers. For generative models: VAE → GANs → Diffusion Models. Each builds foundational concepts for more complex architectures.

What is the difference between implementing Transformers and BERT?

Transformers (Attention Is All You Need) is the base encoder-decoder architecture. BERT uses only the transformer encoder with: (1) Bidirectional attention (sees full context), (2) Masked Language Modeling for pre-training, (3) Next Sentence Prediction task, (4) WordPiece tokenization. BERT is a transformer-based model with specific pre-training objectives.

How do I implement ResNet from scratch in PyTorch?

ResNet implementation steps: (1) Create BasicBlock with two 3x3 convs and skip connection, (2) Create Bottleneck block with 1x1→3x3→1x1 convs for deeper networks, (3) Handle dimension mismatch with 1x1 projection in skip connection, (4) Stack blocks: conv1 (7x7) → layer1 (64 channels) → layer2 (128) → layer3 (256) → layer4 (512) → global avg pool → FC layer.

What are the key components to implement GANs?

GAN implementation requires: (1) Generator: maps noise vector to fake image using transposed convolutions, (2) Discriminator: binary classifier (real vs fake) using convolutions, (3) Adversarial loss: minimax game where generator minimizes log(1-D(G(z))) and discriminator maximizes log(D(x)) + log(1-D(G(z))), (4) Alternating training: train D on real+fake batch, then train G to fool D, (5) Techniques to prevent mode collapse.

How does U-Net differ from regular CNNs?

U-Net is an encoder-decoder with skip connections for pixel-wise segmentation: (1) Contracting path (encoder): conv blocks with max pooling to capture context, (2) Expanding path (decoder): upsampling with transposed convs to enable precise localization, (3) Skip connections: concatenate encoder features to decoder at matching resolutions, preserving spatial information lost during downsampling. The U-shape gives it its name.

Research Papers

gpt-oss

Attention Sinks

Plus

Learned per-head scalar added inside the softmax denominator, giving each attention head an escape valve so weights no longer sum to 1.

YaRN RoPE

Plus

NTK-by-parts inverse frequency scaling that extends RoPE to long contexts.

Sliding Window

Plus

Causal mask that restricts each query to the previous W tokens, used on alternating attention layers.

GQA Attention

Plus

Forward pass combining grouped-query attention, optional sliding window, and learned softmax sinks.

MXFP4 Dequant

Plus

Microscaling FP4: dequantize 4-bit expert weights using a shared per-block FP8 scale.

MoE Router

Plus

Top-k expert routing with softmax taken after the top-k selection, not before.

MoE Forward

Plus

Sparse MoE forward: route, run clamped-SwiGLU experts, weighted-sum the outputs.

Transformer Block

Plus

Pre-norm GPT-OSS block: RMSNorm, GQA attention with sinks plus optional sliding window, residual, RMSNorm, sparse MoE, residual.

Full Forward

Plus

Embedding plus stacked GPT-OSS blocks (alternating attention pattern), final RMSNorm, lm head logits.

Loading architecture visualization...