How do I learn to implement ML papers from scratch?

Start by choosing a paper that matches your interest: Transformers for NLP, ResNet for computer vision, or GANs for generative models. Break the paper into components and implement each as a coding problem. Test your implementation against known results. Build progressively: start with RNN, then LSTM, then Transformers.

Which ML paper should I implement first?

For computer vision beginners: VGG (simple) → ResNet (residual connections). For NLP: RNN → LSTM → Transformers. For generative models: VAE → GANs → Diffusion Models. Each builds foundational concepts for more complex architectures.

What is the difference between implementing Transformers and BERT?

Transformers (Attention Is All You Need) is the base encoder-decoder architecture. BERT uses only the transformer encoder with: (1) Bidirectional attention (sees full context), (2) Masked Language Modeling for pre-training, (3) Next Sentence Prediction task, (4) WordPiece tokenization. BERT is a transformer-based model with specific pre-training objectives.

How do I implement ResNet from scratch in PyTorch?

ResNet implementation steps: (1) Create BasicBlock with two 3x3 convs and skip connection, (2) Create Bottleneck block with 1x1→3x3→1x1 convs for deeper networks, (3) Handle dimension mismatch with 1x1 projection in skip connection, (4) Stack blocks: conv1 (7x7) → layer1 (64 channels) → layer2 (128) → layer3 (256) → layer4 (512) → global avg pool → FC layer.

What are the key components to implement GANs?

GAN implementation requires: (1) Generator: maps noise vector to fake image using transposed convolutions, (2) Discriminator: binary classifier (real vs fake) using convolutions, (3) Adversarial loss: minimax game where generator minimizes log(1-D(G(z))) and discriminator maximizes log(D(x)) + log(1-D(G(z))), (4) Alternating training: train D on real+fake batch, then train G to fool D, (5) Techniques to prevent mode collapse.

How does U-Net differ from regular CNNs?

U-Net is an encoder-decoder with skip connections for pixel-wise segmentation: (1) Contracting path (encoder): conv blocks with max pooling to capture context, (2) Expanding path (decoder): upsampling with transposed convs to enable precise localization, (3) Skip connections: concatenate encoder features to decoder at matching resolutions, preserving spatial information lost during downsampling. The U-shape gives it its name.

Research Papers

GLM-4.5

RMSNorm

Plus

Root Mean Square Norm: stable, no-bias normalization used pre-attention and pre-FFN throughout GLM-4.5.

RoPE Freqs

Plus

Base RoPE inverse-frequency table used by GLM-4.5 attention.

Partial RoPE

Plus

Apply RoPE only to the first rope_dim channels of each head; leave the remaining channels position-free.

QK Norm

Plus

Normalize the query and key with RMSNorm before the dot product to stabilize attention logits.

GQA Attention

Plus

Forward pass combining GQA, partial RoPE, and QK-norm before scaled dot-product attention.

Dense SwiGLU

Plus

SwiGLU FFN used in the dense prefix layer of GLM-4.5 (layers before the MoE blocks).

Group Router

Plus

Group-restricted top-k routing: sigmoid scores, pick topk_group groups first, then top-k experts within those.

Shared Expert

Plus

Always-on expert that processes every token, summed with the routed mixture, with a routed scaling factor.

MoE Forward

Plus

Sparse MoE forward combining group-routed top-k, SwiGLU experts, shared expert, and routed_scaling_factor.

MTP Head

Plus

Multi-Token Prediction head used in GLM-4.5 to predict additional future tokens for speculative decoding.

Transformer Block

Plus

Pre-norm GLM-4.5 block: RMSNorm, QK-norm attention with partial RoPE, residual, RMSNorm, dense-or-MoE FFN, residual.

Full Forward

Plus

Embedding plus stacked GLM-4.5 blocks (1 dense prefix layer, then MoE layers), final RMSNorm, lm_head logits.

Loading architecture visualization...