# Example for Fine Tuning CLIP or BLIP2 for VQA
**URL:**
**Author:**  [@VedaantJain](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FVedaantJain)%5C
**Post date:** [August 15, 2023, 1:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexample-for-fine-tuning-clip-or-blip2-for-vqa%2F50772%2F1 "2023-08-15T01:06:45Z")
Hi,
I wanted to fine tune CLIP and BLIP2 for a VQA task on custom dataset, but I was unsure how to do it. Are there any examples for fine tuning CLIP and BLIP2 for VQA?
Thank you
---
**Author:**  [@dino-chiio](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdino-chiio)%5C
**Post date:** [December 5, 2023, 6:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexample-for-fine-tuning-clip-or-blip2-for-vqa%2F50772%2F2 "2023-12-05T06:24:58Z")
I have implemented a repo [here](https://github.com/dino-chiio/blip-vqa-finetune). I hope this can help.
---
**Author:**  [@swtb](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fswtb)%5C
**Post date:** [February 16, 2024, 12:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexample-for-fine-tuning-clip-or-blip2-for-vqa%2F50772%2F3 "2024-02-16T12:46:53Z")
This uses BLIP rather than BLIP2, no? Any pointers on BLIP-2? The architecture is slightly different
---
**Author:**  [@aditya-nisal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Faditya-nisal)%5C
**Post date:** [April 11, 2024, 4:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexample-for-fine-tuning-clip-or-blip2-for-vqa%2F50772%2F4 "2024-04-11T16:54:03Z")
Hello @swtb , did you find anything for finetuning vqa or blip2 or have implemented anything? Any help will be really appreciated.
---
**Author:**  [@swtb](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fswtb)%5C
**Post date:** [April 12, 2024, 7:22am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexample-for-fine-tuning-clip-or-blip2-for-vqa%2F50772%2F5 "2024-04-12T07:22:10Z")
Yes though I moved on from it. You can find alot of clues from the inputs of the model and outputs of its forward pass.
I’ll be at my pc later, will attach a code snippet from my training loop
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [April 12, 2024, 11:31am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexample-for-fine-tuning-clip-or-blip2-for-vqa%2F50772%2F6 "2024-04-12T11:31:05Z")
Tutorials for fine-tuning BLIP-2 are linked here: [Transformers-Tutorials/BLIP-2 at master · NielsRogge/Transformers-Tutorials · GitHub](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/BLIP-2). These include notebooks for both full fine-tuning (updating all parameters) as well as PEFT (parameter efficient fine-tuning using LoRa).
---
**Author:**  [@wildsky](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwildsky)%5C
**Post date:** [May 24, 2024, 11:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexample-for-fine-tuning-clip-or-blip2-for-vqa%2F50772%2F7 "2024-05-24T11:46:36Z")
Hi, have you implemented the use of lora to fine tune blip2 on top of vqa tasks?
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [May 30, 2024, 7:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexample-for-fine-tuning-clip-or-blip2-for-vqa%2F50772%2F8 "2024-05-30T07:52:23Z")
I’d recommend checking out the [repo](https://github.com/dino-chiio/blip-vqa-finetune) linked above, and then you just need to wrap the BLIP-2 model using PEFT:
```python
from transformers import Blip2ForConditionalGeneration
from peft import get_peft_model, LoraConfig, prepare_model_for_kbit_training
model = Blip2ForConditionalGeneration.from_pretrained("Salesforce/blip2-opt-2.7b", load_in_8bit=True)
lora_config = LoraConfig(
r=8,
lora_alpha=8,
lora_dropout=0.1,
target_modules="...",
init_lora_weights="gaussian",
)
model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
```
Then proceed as usual. The `target_modules` are model-specific.
---
**Author:**  [@NirmalML](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNirmalML)%5C
**Post date:** [October 2, 2024, 2:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexample-for-fine-tuning-clip-or-blip2-for-vqa%2F50772%2F9 "2024-10-02T14:57:13Z")
Can anyone please share the code for Finetuning BLIP-2 for VQA?? (PEFT)
---
**Author:**  [@sashika](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsashika)%5C
**Post date:** [October 14, 2024, 8:51am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexample-for-fine-tuning-clip-or-blip2-for-vqa%2F50772%2F10 "2024-10-14T08:51:56Z")
@NirmalML : were you able to find a suitable code?
---
**Author:**  [@NirmalML](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNirmalML)%5C
**Post date:** [October 14, 2024, 12:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexample-for-fine-tuning-clip-or-blip2-for-vqa%2F50772%2F11 "2024-10-14T12:54:52Z")
Hello @sashika, no not yet. I have a code but I’m not sure if it is correct
---
**Author:**  [@NirmalML](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNirmalML)%5C
**Post date:** [October 14, 2024, 1:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexample-for-fine-tuning-clip-or-blip2-for-vqa%2F50772%2F12 "2024-10-14T13:01:41Z")
I need it urgently, so it would be really kind and helpful if anyone helps me in this
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [October 14, 2024, 1:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexample-for-fine-tuning-clip-or-blip2-for-vqa%2F50772%2F13 "2024-10-14T13:05:42Z")
@not-lain He seems to be in a hurry. Can you think of anyone who might know this code?
> code for Finetuning BLIP-2 for VQA?? (PEFT)
---
**Author:**  [@sashika](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsashika)%5C
**Post date:** [October 15, 2024, 6:18am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexample-for-fine-tuning-clip-or-blip2-for-vqa%2F50772%2F14 "2024-10-15T06:18:45Z")
Could you please share your code?
---
**Author:**  [@VedaantJain](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FVedaantJain)%5C
**Post date:** [October 15, 2024, 6:47am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexample-for-fine-tuning-clip-or-blip2-for-vqa%2F50772%2F15 "2024-10-15T06:47:49Z")
I ended up using BLIP not BLIP2, so:
my complete script for BLIP is here:
>
If looking for a basic snippet,
```auto
encoding = self.processor(images=image, text=question, padding="max_length", truncation=True, return_tensors="pt")
labels = self.processor(text=answer, return_tensors="pt").input_ids
encoding["labels"] = labels
batch = encoding
batch = {k: v.cuda() for k, v in batch.items()}
outputs = model(**batch)
loss = outputs.loss
```
if you are interested in the loss used by the model by default, it is on line 899 on [transformers/src/transformers/models/blip/modeling\_blip\_text.py at main · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/main/src/transformers/models/blip/modeling_blip_text.py#895)
putting the small snippet here:
```auto
if labels is not None:
# we are doing next-token prediction; shift prediction scores and input ids by one
shifted_prediction_scores = prediction_scores[:, :-1, :].contiguous()
labels = labels[:, 1:].contiguous().to(shifted_prediction_scores.device)
loss_fct = CrossEntropyLoss(reduction=reduction, label_smoothing=self.label_smoothing)
lm_loss = loss_fct(shifted_prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))
if reduction == "none":
lm_loss = lm_loss.view(prediction_scores.size(0), -1).sum(1)
```
---
**Author:**  [@VedaantJain](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FVedaantJain)%5C
**Post date:** [October 15, 2024, 6:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexample-for-fine-tuning-clip-or-blip2-for-vqa%2F50772%2F16 "2024-10-15T06:50:57Z")
The above basically calculates over all predictions so every token after the first one not just the answer
but empirically that is fine and it helps if one wants to ask specific kinds of questions as the model is better tuned for the question too.
---
**Author:**  [@bharatgandhi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbharatgandhi)%5C
**Post date:** [February 20, 2025, 12:12am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexample-for-fine-tuning-clip-or-blip2-for-vqa%2F50772%2F17 "2025-02-20T00:12:43Z")
Hey @swtb , can you attach the code?
---
**Author:**  [@VedaantJain](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FVedaantJain)%5C
**Post date:** [February 20, 2025, 12:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexample-for-fine-tuning-clip-or-blip2-for-vqa%2F50772%2F18 "2025-02-20T00:52:29Z")
Hey,
I have attached my code in the discussion already if it helps.
---
**Author:**  [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [February 20, 2025, 12:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexample-for-fine-tuning-clip-or-blip2-for-vqa%2F50772%2F19 "2025-02-20T12:52:53Z")
Closed on February 20, 2025, 12:52pm UTC
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.