Build from first principles

Write FlashAttention Step by Step

Build attention that works on small tiles with a running softmax, prove it matches standard attention exactly, and count the memory traffic that makes it fast.

What is FlashAttention?Why attention is limited by memory, and how tiling avoids the n by n matrix

1Safe Softmax with Masked RowsMake softmax safe

2Online Softmax over ChunksSoftmax one chunk at a time

3Merge Two Partial Attention ResultsCombine partial results

4Tiled Attention Forward PassAttention in tiles

5Which Causal Tiles Can Be SkippedSkip masked tiles

6Count Memory Reads and WritesCount the memory traffic

From Tiles to a KernelSee how your functions become one tiled attention loop

Write FlashAttention in the NotebookBuild it in PyTorch, catch the rescale bug, count the traffic, and finish in Triton