Build from first principles

Train a Tiny CLIP from Scratch

Train a picture encoder and a caption encoder until they agree, use them to search and classify, then catch the shortcut the model takes and fix it with hard negatives.

What is CLIP?Understand two encoders, one shared space, and learning from captions

1Build the Scaled Similarity MatrixCompare pictures and captions in one space

2Write the Symmetric Contrastive LossTrain both encoders to agree

3Measure Retrieval with Recall at KMeasure how well it retrieves

4Classify with Prompts, Zero-ShotClassify with written prompts

5Add Hard Negatives to the Contrastive LossFix the shortcut with hard negatives

6Write the Pairwise Sigmoid LossScore every pair on its own

From Pieces to a Tiny CLIPSee how your functions fit together before training the model

Train, Test, and Fix a Tiny CLIPTrain on drawn shapes, search and classify, prove the shortcut, and fix it