# Stable Diffusion CLIP similarity
**URL:**
**Author:**  [@Cajoek](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FCajoek)%5C
**Post date:** [October 28, 2022, 7:14am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-clip-similarity%2F25118%2F1 "2022-10-28T07:14:21Z")
Hi,
the `StableDiffusionPipeline` uses a `CLIPTextModel` for the text embeddings, can I match that with the corresponding `CLIPVisionModel` to get CLIP test-image similarity of generated images? Which pretrained `CLIPVisionModel` should I use?
Thanks!
---
**Author:**  [@fredguth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffredguth)%5C
**Post date:** [November 1, 2022, 3:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-clip-similarity%2F25118%2F2 "2022-11-01T15:30:12Z")
This is exactly what I am trying to do.
But I am blocked by the fact that the CLIPVisionModel embedding output has a different size than I expected.
In this [minimum working example](https://huggingface.co/spaces/fredguth/mwe_clip/blob/main/mwe.ipynb) you can see that despite `CLIPVisionModel.config.projection_dim == CLIPTextModel.config.projection_dim==768`.
Still, the embeddings present of different shapes:
```auto
CLIPVisionEmbeddings(
(patch_embedding): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14), bias=False)
(position_embedding): Embedding(257, 1024)
)
CLIPTextEmbeddings(
(token_embedding): Embedding(49408, 768)
(position_embedding): Embedding(77, 768)
)
```
I am confused about this and I can’t find much more in the documentation.
---
**Author:**  [@pcuenq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpcuenq)%5C
**Post date:** [November 1, 2022, 4:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-clip-similarity%2F25118%2F3 "2022-11-01T16:50:23Z")
Hello!
You can compute the similarity score between text and image by computing the dot product between the corresponding (normalized) embeddings. In order to do so, you should use the same CLIP Vision model as the one used to retrieve the text embeddings. See for example this snippet here: [transformers/modeling\_clip.py at v4.24.0 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/v4.24.0/src/transformers/models/clip/modeling_clip.py#L1045-L1058). If you invoke that function with multiples images and texts, `logits_per_image` would give you the scores of each image across multiple text prompts, and `logits_per_text` would give you the score about how much each image matches the corresponding text. You can use one image and one prompt and get a score that would be comparable with other image-text pairs.
There’s an usage example here [CLIP](https://huggingface.co/docs/transformers/model_doc/clip#usage) that could be a good starting point 🙂
---
**Author:**  [@fredguth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffredguth)%5C
**Post date:** [November 1, 2022, 5:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-clip-similarity%2F25118%2F4 "2022-11-01T17:19:06Z")
Thanks!
This is what I was looking for:
```nohighlight
with torch.no_grad():
vision_outputs = model.vision_model(pixel_values=inputs.pixel_values)
text_outputs = model.text_model(input_ids=inputs.input_ids)
img_embeds = model.visual_projection(vision_outputs[1])
txt_embeds = model.text_projection(text_outputs[1])
img_embeds = img_embeds / img_embeds.norm(p=2, dim=-1, keepdim=True)
txt_embeds = txt_embeds / txt_embeds.norm(p=2, dim=-1, keepdim=True)
img_embeds.shape, txt_embeds.shape
```
`(torch.Size([1, 768]), torch.Size([1, 768]))`
I was using `transformers` ‘4.23.1’…
---
**Author:**  [@fredguth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffredguth)%5C
**Post date:** [November 1, 2022, 5:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-clip-similarity%2F25118%2F5 "2022-11-01T17:27:17Z")
Ok, I upgraded and `model.vision_model.embeddings, model.text_model.embeddings`are not the same than projection embeddings. I guess I understood it wrongly, but anyway, I can do what I wanted with the projection.
---
**Author:**  [@pcuenq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpcuenq)%5C
**Post date:** [November 1, 2022, 5:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-clip-similarity%2F25118%2F6 "2022-11-01T17:56:43Z")
Oh yes, you are right, you need the projection embeddings 🙂
---
**Author:**  [@Cajoek](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FCajoek)%5C
**Post date:** [December 6, 2022, 5:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-clip-similarity%2F25118%2F7 "2022-12-06T17:07:46Z")
Great that you solved it @fredguth! How did you initalize a working CLIP pipeline from the components of StableDiffusion?
Thanks!