CUDA Kernels

CUDA is NVIDIA's platform for writing programs that run on the GPU. In this plan you write 35 CUDA C++ kernels in six sections, from element-wise operations to matrix operations, reductions, convolutions and ML building blocks, ending with a tiled matrix multiply.

Basic C or C++ syntax and comfort with arrays and loops. No GPU experience is needed: the first section introduces threads and blocks with simple element-wise kernels.

1. Kernel foundations: vector addition and subtraction, and activations such as ReLU, sigmoid, tanh, softmax, GELU and Swish.
2. Matrix operations: addition, transpose, multiplication, outer and Hadamard products, matrix-vector products, LayerNorm and RMSNorm.
3. Reductions: dot product, sum, L1 and L2 normalization, mean and variance, min, max, argmin and argmax.
4. Convolutions: 1D, 2D and 3D convolution and 2D max pooling.
5. ML primitives: cross-entropy loss, dropout, scaled dot-product attention and cosine similarity.
6. Performance: tiled matrix multiplication using shared memory.

Each kernel runs on real GPU hardware and is checked against correctness tests. Each section ends with a short quiz on the concepts behind the kernels.

Write the same kernels at a higher level with the Triton study plan, then apply them to model serving in the Inference Engineering study plan.