# Custom VLM - Swapping a vision encoder from a VLM **URL:** **Author:** ![mvish7](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mvish7/32/43604_2.png) [@mvish7](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmvish7)%5C **Post date:** [March 19, 2025, 9:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-vlm-swapping-a-vision-encoder-from-a-vlm%2F146421%2F1 "2025-03-19T09:58:38Z") Hi, can you provide some support on How can I create a custom VLM by changing the Vision encoder (and projector/connector module) from a VLM? E.g. I would like to replace SigLip1 encoder from PaliGemma2 by SigLip2 or use SigLip2 encoder with Qwen2.5 vl - 3B model. In any case, I want to retain the language model part from VLM and LMHead. I would like to know if this is natively possible with transformers or do I have to fork and patch? also How checkpoint loading will look like when I create a custom VLM by replacing the vision encoder and projector module? Thanks, Vishal ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C **Post date:** [March 19, 2025, 11:14am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-vlm-swapping-a-vision-encoder-from-a-vlm%2F146421%2F2 "2025-03-19T11:14:07Z")
That’s quite difficult… I’ll leave a link here that might be helpful. > also How checkpoint loading will look like when I create a custom VLM by replacing the vision encoder and projector module? Simply like one of these: - model = YourNewModelClass.from\_pretrained(“your\_id/your\_checkpoint”, trust\_remote\_code=True) - model = YourNewModelClass.from\_pretrained(“./your\_local\_checkpoint”) ### References > **[Customizing models](https://huggingface.co/docs/transformers/main/en/custom_models)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. [https://medium.com/@uthmant14/building-a-custom-vision-language-model-with-hugging-face-7d51099bfbcb](https://medium.com/@uthmant14/building-a-custom-vision-language-model-with-hugging-face-7d51099bfbcb) > **[Welcome to the Community Computer Vision Course - Hugging Face Community...](https://huggingface.co/learn/computer-vision-course/unit0/welcome/welcome)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. > **[SmolVLM - small yet mighty Vision Language Model](https://huggingface.co/blog/smolvlm)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. * * * To create a custom Vision-Language Model (VLM) by replacing the vision encoder and projector module while retaining the language model and LMHead, you can leverage the Hugging Face Transformers library, which provides tools and classes to support such modifications. Below is a guide on how to approach this: * * * ### **Approach** The idea is to create a new VLM by combining a custom vision encoder (e.g., SigLIP2) with an existing language model (e.g., Qwen2.5 VL-3B). You can achieve this without forking the Transformers repository by using the provided classes and methods to initialize and combine models. * * * ### **Process** 1. **Load the Vision Encoder and Language Model** 2. **Modify the Vision Encoder** 3. **Combine the Vision Encoder and Language Model** 4. **Save the Custom VLM** 5. **Load the Custom VLM** * * * ### **Code Examples** #### Example 1: Replacing the Vision Encoder The following example demonstrates how to replace the vision encoder in a VLM with a custom encoder: ```python from transformers import VisionEncoderDecoderModel import torch.nn as nn # Define the new vision encoder class CustomVisionEncoder(nn.Module): def __init__ (self): super(). __init__ () # Define your custom vision encoder layers here # Example: self.conv = nn.Conv2d(3, 64, 3, padding=1) self.fc = nn.Linear(64 * 224 * 224, 512) def forward(self, pixel_values): x = self.conv(pixel_values) x = x.view(x.size(0), -1) x = self.fc(x) return x # Load the language model language_model = AutoModelForCausalLM.from_pretrained("your_language_model_path") # Initialize the custom VLM custom_vlm = VisionEncoderDecoderModel( encoder=CustomVisionEncoder(), decoder=language_model ) ``` #### Example 2: Using a Custom Vision Encoder with a Projector Module The following example demonstrates how to replace the vision encoder and projector module in a VLM: ```python from transformers import VisionEncoderDecoderModel import torch.nn as nn # Define the new vision encoder with a projector module class CustomVisionEncoderWithProjector(nn.Module): def __init__ (self, projection_dim): super(). __init__ () # Define your custom vision encoder layers here self.conv = nn.Conv2d(3, 64, 3, padding=1) self.fc = nn.Linear(64 * 224 * 224, 512) self.projector = nn.Linear(512, projection_dim) def forward(self, pixel_values): x = self.conv(pixel_values) x = x.view(x.size(0), -1) x = self.fc(x) x = self.projector(x) return x # Load the language model language_model = AutoModelForCausalLM.from_pretrained("your_language_model_path") # Initialize the custom VLM with the new encoder and projector custom_vlm = VisionEncoderDecoderModel( encoder=CustomVisionEncoderWithProjector(projection_dim=512), decoder=language_model ) ``` * * * ### **Handling Checkpoint Loading** When you create a custom VLM, you can save and load checkpoints as follows: #### Saving the Model ```python # Save the custom VLM custom_vlm.save_pretrained("path_to_save") ``` #### Loading the Model ```python # Load the custom VLM loaded_model = VisionEncoderDecoderModel.from_pretrained("path_to_save") ``` * * * ### **Conclusion** By utilizing the flexibility of the Transformers library and the `VisionEncoderDecoderModel` class, you can create a custom VLM by replacing the vision encoder and projector module while retaining the language model and LMHead. This approach does not require forking the repository, as the library’s existing methods and classes provide the necessary support.