What is a Vision Transformer?
Introduction
Project introduction
What is a Vision Transformer?
Transformers were built for text. They read a sequence of word vectors and let every word look at every other word. In 2020 a Google team showed that the same model works on images, once an image is turned into a sequence. They cut it into square patches, flattened each patch into a vector, and called the paper "An Image is Worth 16x16 Words".
That model is the Vision Transformer, or ViT. Apart from cutting the image into patches, it knows nothing about images. It does not know that neighbouring pixels belong together, or that a shoe is still a shoe when it moves a little. It has to learn all of that from data.
Why compare it with a CNN?
A convolutional network (CNN) is built for images. Each of its filters looks at a small window of neighbouring pixels and the same filter slides over the whole image. These built-in assumptions are called inductive biases, and they are very useful when you have little data.
The original ViT trailed strong CNNs when trained on the 1.3 million ImageNet images, matched them with 14 million, and only beat them with 300 million. In this project you build a tiny ViT, race it against a tiny CNN on a few thousand clothing photos, and measure exactly what the missing assumptions cost.
An image as a sequence of patches
16 patches + 1 class tokenToken sequence, in reading order
Patch 1 of 16 joins the sequence.
What you will build
01
Images as sequences
Cut images into patches, embed them, and add a class token and positions.
02
Attention over patches
Write attention and a transformer block, then measure how far each head looks.
03
An honest race
Train a ViT and a CNN with the same budget across seeds, then add augmentation and data.
Start with patches
Cut a batch of images into patch vectors with one reshape and one permute.