# Fine tuning image transformer on higher resolution
**URL:**
**Author:**  [@moshel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmoshel)%5C
**Post date:** [September 6, 2022, 11:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-image-transformer-on-higher-resolution%2F22623%2F1 "2022-09-06T11:10:37Z")
I would like to finme tune a transformer on images 512x512. I was looking at the huggingpics notebook as a starting point but failed to modify it to train on images with size different from 224x224.
I tried adding the “size” to ViTFeatureExtractor.from\_pretrained (unknown keyword) and to ViTForImageClassification.from\_pretrained (passed, but when it reached the trainer.fit fails with “ValueError: Input image size (512_512) doesn’t match model (224_224).”).
While i know how to do that in plain vanilla keras and pytorch, the hugginingface is just too user friendly for me!
Any help will be greatly appreciated.
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [September 7, 2022, 12:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-image-transformer-on-higher-resolution%2F22623%2F2 "2022-09-07T12:29:54Z")
Hi,
Yes you can achieve that, by passing `interpolate_pos_encoding=True` to the forward of the model ([docs](https://huggingface.co/docs/transformers/model_doc/vit#transformers.ViTForImageClassification.forward.interpolate_pos_encoding)).
Explanation: so the Vision Transformer (ViT) was pre-trained on images of resolution 224x224. For instance, [google/vit-base-patch16-224 · Hugging Face](https://huggingface.co/google/vit-base-patch16-224) used images of size 224x224 and a patch resolution of 16x16. This means that, when feeding an image to the model, one gets (224/16)^2 = 196 patch tokens, which are fed to the Transformer encoder. As one also adds a special CLS token at the beginning, one actually feeds 197 tokens to it.
The model adds position embeddings to each of those patch tokens, to make the model “know” the order of these tokens. However, one only has an embedding matrix containing 197 positions during pre-training.
Suppose you’d like to fine-tune the model on a resolution of 512x512. With a patch resolution of 16x16, one has (512/16)^2 = 1024 patch tokens, and with the additional CLS token, this would mean 1025 tokens! Which means we need 1025 position embeddings.
So what usually happens is **interpolation of the pre-trained position embeddings** , to basically turn the embedding matrix from (197, 768) to (1025, 768) - assuming each position embedding is a vector of size 768.
See also section 3.2 of the [paper](https://arxiv.org/abs/2010.11929).
---
**Author:**  [@moshel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmoshel)%5C
**Post date:** [September 7, 2022, 11:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-image-transformer-on-higher-resolution%2F22623%2F3 "2022-09-07T23:34:46Z")
Thank you so much! now its working!
IMHO, as this is definitely not trivial and different to other models, that perhaps a small section on finetuning should be added? as well as feature extraction (it seems to be different, much smaller vectors to, say efficientnet b7 that has 2580 or something and vit has only 768). Also the pooling mechanism is different, the the unusual max and avg pooling does not work (which makes sense as it is embeddings of patches and not actually image patches). But, all this is unexplained.
Thank you again for your help! much appreciated!
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [September 8, 2022, 9:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-image-transformer-on-higher-resolution%2F22623%2F4 "2022-09-08T09:48:44Z")
Hi,
Yes definitely, let me open a PR today to add this!
And yes ViT is very much like BERT 😉 it outputs a vector of size 768 for each “patch” (which can be seen as each “word”), whereas a model like ResNet outputs a “feature map” of shape (batch\_size, num\_channels, height, width).
---
**Author:**  [@moshel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmoshel)%5C
**Post date:** [September 10, 2022, 2:41am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-image-transformer-on-higher-resolution%2F22623%2F5 "2022-09-10T02:41:18Z")
Umm i am still doing something wrong, I feel, as I can train really large batches (over 128) which shouldn’t be possible even with 24gb of gpu memory and results are about the same as with the 224
I’ll detail what I do:
This is the model i use:
```auto
def get_model(size):
feature_extractor = ViTFeatureExtractor.from_pretrained('google/vit-base-patch32-384', size = (512,512))
model = ViTForImageClassification.from_pretrained(
'google/vit-base-patch32-384',
num_labels=2,
¦ ignore_mismatched_sizes=True
)
return model, feature_extractor
```
this is the training step in the pl LightningModule class:
```auto
def training_step(self, batch, batch_idx):
¦ outputs = self(**batch, interpolate_pos_encoding=True)
¦ self.log(f"train_loss", outputs.loss)
¦ return outputs.loss
```
the rest is basically the same as the huggingpics notebook. what am I missing?
---
**Author:**  [@jorgeili](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjorgeili)%5C
**Post date:** [September 27, 2022, 7:16am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-image-transformer-on-higher-resolution%2F22623%2F6 "2022-09-27T07:16:19Z")
Hi,
I’m findind hard to achieve to set the interporlate\_pos\_encoding = True by redefining the model’s forward method.
Could you make a brief step by step case on how to do so in order to train the model, and not just make a forward pass?
I thought it was just about modifying the model.config which feeds the module parameters, as one can set the output\_attentions = True per example, but I see is not the same case for the inteporlate\_pos\_encoding.
Thank you!
---
**Author:**  [@jorgeili](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjorgeili)%5C
**Post date:** [September 27, 2022, 7:39am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-image-transformer-on-higher-resolution%2F22623%2F7 "2022-09-27T07:39:39Z")
I fine tune the model by using the Trainer build class from transformers, not directly by calling the forward method, so I’m not finding the way to set the interpolate\_pos\_encoding to true in that case.
---
**Author:**  [@Adhil1123](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAdhil1123)%5C
**Post date:** [February 12, 2024, 6:39am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-image-transformer-on-higher-resolution%2F22623%2F8 "2024-02-12T06:39:11Z")
ValueError: mean must have 4 elements if it is an iterable, got 3
my program is showing above error how to solve it
---
**Author:**  [@gabrielmotablima](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgabrielmotablima)%5C
**Post date:** [February 15, 2024, 4:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-image-transformer-on-higher-resolution%2F22623%2F9 "2024-02-15T16:58:56Z")
Same error to me while trying to train an image captioning model (labels size 32, pixel values size [3, 224, 224])
```auto
ValueError: mean must have 4 elements if it is an iterable, got 3
```
---
**Author:**  [@gabrielmotablima](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgabrielmotablima)%5C
**Post date:** [March 6, 2024, 3:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-image-transformer-on-higher-resolution%2F22623%2F10 "2024-03-06T03:01:13Z")
@Adhil1123 I got the solution for `ValueError: mean must have 4 elements if it is an iterable, got 3`
My images were in CMYK mode, so they had 4 channels intead of 3. I had to convert them to RGB with `img.convert('RGB')` to run the code successfuly .
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [May 1, 2024, 4:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-image-transformer-on-higher-resolution%2F22623%2F11 "2024-05-01T16:09:04Z")
> [@jorgeili](#):
>
> I’m findind hard to achieve to set the interporlate\_pos\_encoding = True by redefining the model’s forward method.
Hi one can achieve this as explained here: [Fine-tuning ViT with more patches/higher resolution - #4 by mohotmoz](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-vit-with-more-patches-higher-resolution%2F18731%2F4)
---
**Author:**  [@moshel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmoshel)%5C
**Post date:** [May 1, 2024, 7:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-image-transformer-on-higher-resolution%2F22623%2F12 "2024-05-01T19:23:21Z")
This is a really good post and actually came at the right time, when I was reconsidering vit. Thank you!