From Patches to a Vision Transformer
Introduction
Project introduction
From Patches to a Vision Transformer
You have written every piece a Vision Transformer needs. These are cutting images into patches, embedding them with a class token and positions, attention, a pre-norm transformer block, and a way to measure how far each attention head looks.
In the notebook these pieces become a tiny ViT with about 20,000 parameters. You train it on Fashion-MNIST clothing photos next to a tiny CNN of the same size, with the same data, the same number of training steps, and three random seeds each.
What you will find out
On 1000 training images the CNN wins clearly, and the ViT gets nearly every training image right while doing much worse on new ones. You will measure why. No attention head learns to look at nearby patches, and a ViT without positions cannot even tell an image from its shuffled patches.
Then you try three fixes. Augmentation stops the memorizing but does not help at this small budget, as the "How to train your ViT" study warns for small, short runs. A penalty that pushes the heads toward nearby patches makes them look nearby, yet accuracy barely moves. More data helps the ViT more than the CNN and narrows the gap, but the CNN stays clearly ahead.
An image as a sequence of patches
16 patches + 1 class tokenToken sequence, in reading order
Patch 1 of 16 joins the sequence.
How the notebook is organized
01
Build and race
Assemble the ViT and the CNN and train both on 1000 images.
02
Explain the gap
Measure attention distance and shuffle patches to see what the ViT has not learned.
03
Help the ViT
Try augmentation, local attention and 8 times more data, then decide which differences are real.
Train a tiny ViT against a CNN
Run the whole investigation in the browser, about 45 minutes.