Build from first principles

Build a Tiny Vision Transformer

Turn images into sequences of patches, build attention and a transformer block from scratch, and find out honestly when a small CNN still beats your Vision Transformer.

What is a Vision Transformer?See how an image becomes a sequence a transformer can read

1Cut Images into Patch VectorsAn image as a sequence of patches

2Embed Patches with a Class Token and PositionsPatch tokens, class token, positions

3Scaled Dot-Product Attention over PatchesLet every patch look at every patch

4Run One Pre-Norm Transformer BlockOne transformer block

5Measure How Far Each Attention Head LooksHow far each head looks

From Patches to a Vision TransformerSee how your functions fit together before training

Train a Tiny ViT Against a CNNTrain both on clothing photos, measure where attention looks, shuffle patches, and add data