# Use custom text encoder in CLIP
**URL:**
**Author:**  [@ThreeBlessings](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FThreeBlessings)%5C
**Post date:** [January 22, 2024, 1:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fuse-custom-text-encoder-in-clip%2F70090%2F1 "2024-01-22T13:21:17Z")
Hi guys!
I want to train a CLIP model for Ukrainian and would like to replace a text encoder with pretrained Ukrainian text encoder. I found a couple of discussions about CLIP implemented in Flax for the [Spanish](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fimage-captioning-for-spanish-with-pre-trained-vision-and-text-model%2F7084) and [Korean](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkoclip-pretraining-clip-on-korean%2F7280) languages and a research papers about replacing CLIP text encoder for other languages (AltCLIP), but they all kinda reimplement CLIP for that.
Is there an easy way to initialize a CLIPModel but with custom text encoder (available on hf hub)? Same question for replacing the tokenizer in the CLIPProcessor.
I would like something like this:
```auto
model = CLIPModel(text_encoder=my_text_encoder, image_encoder=my_image_encoder)
processor = CLIPProcessor(tokenizer=my_tokenizer, image_processor=my_image_processor)
```
---
**Author:**  [@syrineM](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FsyrineM)%5C
**Post date:** [January 22, 2024, 1:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fuse-custom-text-encoder-in-clip%2F70090%2F2 "2024-01-22T13:57:29Z")
Hi, the solution i found is to use VisionTextDualEncoder class rather than CLIPModel class of transformers library where you can customize the vision and text encoders : " This model can be used to align the vision-text embeddings using CLIP like contrastive image-text training and then can be used for zero-shot vision tasks such image-classification or retrieval."
Not sure all text encoders architectures are compatible but worth a try !
---
**Author:**  [@ThreeBlessings](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FThreeBlessings)%5C
**Post date:** [January 22, 2024, 2:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fuse-custom-text-encoder-in-clip%2F70090%2F3 "2024-01-22T14:07:00Z")
Thank you so much! This looks like exactly what I need! Another thing I didn’t mention was that it’s actually for SigLIP model and as I see `VisionTextDualEncoder` only uses CLIP loss for now. I will create a PR to make it work with SigLIP loss as well 🙂
---
**Author:**  [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [January 23, 2024, 2:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fuse-custom-text-encoder-in-clip%2F70090%2F4 "2024-01-23T02:07:04Z")
Closed on January 23, 2024, 2:07am UTC
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.