Compute symmetric CLIP contrastive loss for B paired image and text embeddings. Matching pairs occupy the same row. Normalize each embedding vector x using
x=max(∥x∥2,10−12)x.Here, the norm is Euclidean; this normalization leaves a zero vector at zero. With image vector v, text vector t, and positive temperature τ, define the similarity matrix
Sij=τviTtj.The symmetric loss is
L=−2B1i=1∑B[log∑j=1BeSijeSii+log∑j=1BeSjieSii].Indices i and j identify pairs; log is the natural logarithm. Each directional accuracy is the fraction of rows or columns whose highest similarity selects its matched diagonal entry. Ties select the lowest index.
Return a Python dictionary of PyTorch tensors: loss, image_to_text_accuracy, and text_to_image_accuracy are scalars; similarities has shape (B, B). All outputs preserve the embedding dtype and device.
Input: image_embeddings = [[1,0],[0,1]], text_embeddings = [[1,0],[0,1]], temperature = 1
Output: {"loss":0.313262,"similarities":[[1,0],[0,1]],"image_to_text_accuracy":1,"text_to_image_accuracy":1}
Explanation: Each diagonal cosine similarity exceeds its off-diagonal alternative by one.
Input: image_embeddings = [[3,4]], text_embeddings = [[6,8]], temperature = 0.5
Output: {"loss":0,"similarities":[[2]],"image_to_text_accuracy":1,"text_to_image_accuracy":1}
Input: image_embeddings = [[1,0],[1,1]], text_embeddings = [[1,0],[0,1]], temperature = 1
Output: {"loss":0.491157,"similarities":[[1,0],[0.707107,0.707107]],"image_to_text_accuracy":0.5,"text_to_image_accuracy":1}
torch.nn.functional.normalize supports a lower bound for the norm denominator.
torch.nn.functional.cross_entropy accepts a logit matrix and class-index targets.
Tensor.argmax returns the first index when maximum values are tied.
Accepts: array
Accepts: array
Accepts: number
Compute symmetric CLIP contrastive loss for B paired image and text embeddings. Matching pairs occupy the same row. Normalize each embedding vector x using
x=max(∥x∥2,10−12)x.Here, the norm is Euclidean; this normalization leaves a zero vector at zero. With image vector v, text vector t, and positive temperature τ, define the similarity matrix
Sij=τviTtj.The symmetric loss is
L=−2B1i=1∑B[log∑j=1BeSijeSii+log∑j=1BeSjieSii].Indices i and j identify pairs; log is the natural logarithm. Each directional accuracy is the fraction of rows or columns whose highest similarity selects its matched diagonal entry. Ties select the lowest index.
Return a Python dictionary of PyTorch tensors: loss, image_to_text_accuracy, and text_to_image_accuracy are scalars; similarities has shape (B, B). All outputs preserve the embedding dtype and device.
Input: image_embeddings = [[1,0],[0,1]], text_embeddings = [[1,0],[0,1]], temperature = 1
Output: {"loss":0.313262,"similarities":[[1,0],[0,1]],"image_to_text_accuracy":1,"text_to_image_accuracy":1}
Explanation: Each diagonal cosine similarity exceeds its off-diagonal alternative by one.
Input: image_embeddings = [[3,4]], text_embeddings = [[6,8]], temperature = 0.5
Output: {"loss":0,"similarities":[[2]],"image_to_text_accuracy":1,"text_to_image_accuracy":1}
Input: image_embeddings = [[1,0],[1,1]], text_embeddings = [[1,0],[0,1]], temperature = 1
Output: {"loss":0.491157,"similarities":[[1,0],[0.707107,0.707107]],"image_to_text_accuracy":0.5,"text_to_image_accuracy":1}
torch.nn.functional.normalize supports a lower bound for the norm denominator.
torch.nn.functional.cross_entropy accepts a logit matrix and class-index targets.
Tensor.argmax returns the first index when maximum values are tied.
Accepts: array
Accepts: array
Accepts: number
