From Pieces to a Tiny CLIP
Introduction
Project introduction
From Pieces to a Tiny CLIP
You have written every piece a CLIP model needs: the scaled similarity matrix, the symmetric contrastive loss, recall at k to measure search, zero-shot prediction from prompts, a loss with hard negatives, and the pairwise sigmoid loss.
In the notebook these pieces train a real model. Two small encoders learn from 6,000 drawn pictures of colored shapes and their captions, in about 20 seconds.
What you will see
After training, a picture finds its exact caption among 400 unseen ones about 80 percent of the time, and the model sorts pictures by shape using only prompts you write. Then you test it with captions that use the same words but swap the colors, and it is right only about 3 times in 4, no better than when it chooses among all 400 captions.
More of the same training helps only a little, to about 80 percent. Training on the same batches with the color swaps added as hard negatives raises that score to about 96 percent, and the result holds across random seeds.
One batch, every pair scored
training 0%How the notebook is organized
01
Build and train
Look at the data, score every pair, write the loss, and train for 600 steps.
02
Use it
Search captions with pictures and classify with prompts.
03
Catch and fix
Prove the shortcut with swapped captions, then fix it with hard negatives and compare fairly.
Train, test, and fix a tiny CLIP
Run the whole investigation in the browser, about 45 minutes.