CUDA Practice Problems: From Vector Add to Attention

The best way to practise CUDA is to solve kernel problems in a deliberate order, starting with one thread per element and building up to reductions, shared-memory tiling, convolutions and attention. TensorTonic's CUDA Kernels study plan has 35 such problems that you solve in the browser and run on real cloud GPUs, so you do not need your own NVIDIA card, and this guide orders all of them into a learning path.

Why was TensorTonic built for practising CUDA?

I built TensorTonic because I wanted to learn GPU programming myself and could not get a working CUDA setup on my own laptop, which is a Mac. Rather than buying an NVIDIA machine, I built an operating-system-agnostic platform where your code runs in a sandbox that already has CUDA installed on a real GPU, and I then worked through these same problems on it.

The problem it solves is the one that stops most people before they write their first kernel: installing drivers and the toolkit, finding compatible hardware, and paying for a GPU just to find out whether CUDA is for you. On TensorTonic you can open a problem on any laptop, write the kernel in the browser and see it run against tests on a real GPU within seconds, so all of your time goes into learning how kernels work rather than into setup.

What should you know before practising CUDA?

Before practising CUDA you should be comfortable with basic programming in C or C++, index arithmetic, simple linear algebra, and the idea that a GPU runs a very large number of small threads at once, whereas a CPU runs a few threads quickly. Everything else can be learned while you solve problems.

• CUDA kernels are written in C++, so pointers, arrays and function signatures should already feel familiar to you.
• Most kernels turn a thread's position into an array index, and two-dimensional problems use row-major indexing, where the element at a given row and column sits at row times width plus column.
• Dot products, matrix multiplication and transposes come up constantly, so it helps to know what they compute before you parallelise them.

Thread blocks, grids, launch configuration, shared memory, synchronisation, memory access patterns and the reasons reductions need care are all easier to absorb when a failing test forces you to use them, so you do not need to read a CUDA textbook before you start.

In what order should you practise CUDA problems?

Practise CUDA problems in eight stages, ordered by the skill each one builds: first kernels, elementwise activations, two-dimensional indexing, reductions, matrix multiplication with tiling, numerically stable row-wise kernels, convolutions, and finally ML primitives and attention. This regroups the study plan's six sections, so softmax, for example, comes after reductions because it depends on them.

Stage 1: Your first kernels

Start with Vector Addition and Vector Subtraction, which both ask for one thread per output element with a bounds check. They teach the basic CUDA execution model, in which threads map to elements, blocks group those threads and the host launches the kernel, and they show why the bounds check matters: launch sizes are usually rounded up, so some threads have no element to process.

Stage 2: Elementwise math and activation functions

Next, solve ReLU, Leaky ReLU, Sigmoid, Tanh, GELU and Swish. The thread-per-element pattern stays the same, so you practise it on real ML functions while learning device math functions such as exponentials, tanh and the error function, along with the branching that ReLU needs. Pay attention to how numeric precision affects the result, and not only to whether the kernel runs.

Stage 3: Two-dimensional indexing

Move to matrices with Matrix Addition, Hadamard Product, Outer Product and Matrix Transpose, which use a two-dimensional grid with row-major, bounds-checked indexing. Transpose is the most instructive of the four, because each element has a simple destination but the reads and the writes cannot both be contiguous, which is your first real encounter with memory access patterns.

Stage 4: Reductions

A reduction collapses many values into one, which breaks the model in which every thread works alone, because threads now have to combine their results. Practise it with Sum of Array, Max of Array, Min of Array, Argmax, Argmin, Dot Product, Mean and Variance, L1 Normalization and L2 Normalization.

Several of these are multi-block reductions, in which results from each block must be combined into a single value. Argmax and argmin add a tie-breaking rule that picks the lowest index, and the normalisation problems chain a reduction with a second pass that divides every element by the result. Watch for race conditions when threads write to the same location, and for the small differences that floating-point addition order introduces.

Stage 5: Matrix multiplication and shared-memory tiling

Work through Matrix-Vector Multiplication, Matrix Multiplication and Tiled Matrix Multiplication in that order. The first two use direct mappings, with one thread per output row or per output element, while the tiled version loads blocks of the inputs into shared memory with synchronised tile loads and handles partial tiles at the edges. Write the simple version first and then compare it with the tiled one, because tiling does not win at every input size.

Stage 6: Numerically stable and row-wise kernels

Now combine reductions with elementwise work in Softmax, Layer Normalization and RMS Normalization. Softmax is stable only if you subtract the row maximum first, so it needs both a maximum reduction and a sum reduction. Layer normalization is a fused row-wise kernel that computes the mean and variance in shared memory before applying scale and bias, and RMS normalization uses a sum-of-squares reduction with a learnable scale. These three are the building blocks of transformer models.

Stage 7: Convolution and pooling

Practise 1D Convolution, 2D Convolution, 3D Convolution and Max Pool 2D. In the one-dimensional problem each thread produces one output element and accumulates over the kernel in registers. As you move to two and three dimensions the indexing grows and the same input values are read many times, which makes data reuse and memory traffic the main questions to think about.

Stage 8: ML primitives and attention

Finish with Cross-Entropy Loss, Dropout, Cosine Similarity and Scaled Dot-Product Attention. The attention problem asks for single-head attention with a stable row-wise softmax and value aggregation, so it reuses almost everything from the earlier stages: a matrix multiplication, a stable softmax and a second matrix multiplication.

Fused attention kernels such as FlashAttention go one step further by processing attention in tiles, so the full score matrix is never written to global memory. Solving the plain version first shows you exactly what a fused version improves, and you can then write FlashAttention forward in Triton or read how FlashAttention works.

How should you practise each CUDA problem?

Practise each CUDA problem by first writing the simplest correct kernel, then testing it on awkward input sizes, and only then optimising one change at a time while benchmarking each change. A slow kernel that passes every test is a better starting point than a clever one that fails.

1. Understand the reference operation well enough to know what the output should be before you think about threads.
2. Decide what each thread computes, whether that is one element, one row or one tile.
3. Write the simplest kernel that produces the correct result.
4. Check correctness on edge cases, especially sizes that do not divide evenly into blocks and very small inputs, which catch most bugs.
5. Inspect memory access by asking which threads read neighbouring addresses and which values are read more than once.
6. Optimise one thing at a time, remembering that more threads or higher occupancy do not automatically make a kernel faster.
7. Benchmark every change across several input sizes, and across different GPUs, because an optimisation stays a hypothesis until the timings confirm it.
8. Write down what changed and why, so that each solved problem becomes knowledge you can reuse.

Do you need your own GPU to practise CUDA?

No, you do not need your own NVIDIA GPU, CUDA toolkit or driver setup to practise CUDA on TensorTonic. When you submit a kernel from the browser, it is compiled and run against correctness tests on real cloud GPUs, and the results come back to the same page.

You can choose which GPU runs your kernel from six options: an NVIDIA T4, RTX 4090, RTX 5090, RTX PRO 6000, H100 or H200. CUDA kernel problems are included in the free plan, so the whole path in this guide is available without paying.

Running kernels in the browser removes setup, drivers and hardware cost, which is what usually stops people from starting. Installing the CUDA toolkit and building projects on your own machine is a separate skill that you can pick up later, and the concepts you practise here carry over to it directly.

How do you benchmark a CUDA kernel?

Benchmark a CUDA kernel by timing it over several input sizes, repeating the measurements, and comparing it with a reference implementation on the same GPU. Small inputs mostly measure launch overhead, while large inputs reveal whether the kernel is limited by memory bandwidth or by compute.

A vector addition is typically bandwidth-bound, while a large, well-optimised matrix multiplication is limited more by compute throughput. Because timings vary between runs, trust consistent gaps rather than small differences.

After a successful submission, TensorTonic shows your kernel's timings across multiple input sizes, compares each one with the reference implementation on the same GPU, and keeps results from different GPUs side by side for the same code. That makes it easy to see when a tile size tuned for one GPU does not suit another.

What should you learn after these CUDA problems?

After the CUDA path, the natural next steps are writing the same kernels in Triton, applying them to LLM training and serving, and implementing fused attention.

• The Triton Kernels study plan covers block-level GPU kernels written in Python, and solving a problem in both CUDA and Triton is a good way to see what each one abstracts away.
• Language Modeling from Scratch uses CUDA and Triton among its tools while building a modern language model stack.
• The Inference Engineering study plan covers attention variants, KV caching, quantization and serving for large language models.
• Once your attention kernel passes, move on to FlashAttention forward to practise tiled, fused attention.

The easiest place to begin is Vector Addition, or you can open the full CUDA Kernels study plan to see all 35 problems.

Related guides

• LeetCode for machine learning: where to practice ML coding
• How to prepare for an ML engineer interview
• Is TensorTonic free? Free, Pro and Plus compared