What is CLIP?
Introduction
Project introduction
What is CLIP?
CLIP is a pair of networks trained together: one reads pictures, the other reads text. Each turns its input into a list of numbers, an embedding, and both embeddings live in the same space. A picture and a sentence that describes it should point in the same direction there.
CLIP learns this from pictures and their captions alone, with no class labels. In every training batch it scores each picture against each caption and is rewarded when every picture picks out its own caption, and every caption its own picture. OpenAI trained the original on 400 million pairs from the internet.
Why build a tiny one?
The ideas behind CLIP fit in a few lines of code: cosine similarity, a temperature, and a loss that reads the diagonal of a matrix. Building them yourself on small drawn pictures of colored shapes makes each idea visible, and the whole model trains in about 20 seconds on a CPU.
The small world also lets you test the model precisely. You will find that it searches well and classifies with written prompts, yet often mixes up which color belongs to which shape. Researchers found the same weakness in real CLIP models, and you will fix it the way they did.
One batch, every pair scored
training 0%What you will build
01
Two encoders, one space
Score every picture against every caption and train with the symmetric contrastive loss.
02
Search and classify
Measure retrieval with recall at k and classify pictures with nothing but written prompts.
03
Catch and fix a shortcut
Swap the colors in a caption, see how often the model is fooled, and fix it with hard negatives.
Start with the similarity matrix
Score every picture against every caption with cosine similarity and a temperature.