# SAM image size for fine-tuning
**URL:**
**Author:**  [@deppen8](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdeppen8)%5C
**Post date:** [May 12, 2023, 3:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsam-image-size-for-fine-tuning%2F39666%2F1 "2023-05-12T15:01:40Z")
I am trying to fine-tune the Segment Anything (SAM) model following the recently-posted [demo notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/SAM/Fine_tune_SAM_(segment_anything)_on_a_custom_dataset.ipynb) (thanks, @nielsr and @ybelkada !).
I am trying to use 1024x1024 pixel images and masks, but when I try to calculate the loss with `loss = seg_loss(predicted_masks, ground_truth_masks.unsqueeze(1))`, I get an error from `monai`:
`AssertionError: ground truth has different shape (torch.Size([1, 1, 1024, 1024])) from input (torch.Size([1, 1, 256, 256]))`
Obviously, somewhere in the `model()` call, the input 1024x1024 tensor is getting downsampled to 256x256. In the example notebook, the input image is 256x256 already, so the mask is as well.
I am wondering what is the best way to handle this. Should I simply downsample my masks before calculating the loss or is there some parameter I can change so that I can use 1024x1024 masks?
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [May 13, 2023, 9:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsam-image-size-for-fine-tuning%2F39666%2F2 "2023-05-13T09:17:58Z")
Hi,
Thanks for your interest in my notebook! You could use torch.nn.functional.interpolate to interpolate the predicted masks to the appropriate size before calculating the loss:
```auto
from torch import nn
predicted_masks = nn.functional.interpolate(predicted_masks,
size=(1024, 1024),
mode='bilinear',
align_corners=False)
```
This is also used in the postprocessing method of `SamImageProcessor`.
---
**Author:**  [@gobleanMW](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FgobleanMW)%5C
**Post date:** [June 5, 2023, 12:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsam-image-size-for-fine-tuning%2F39666%2F3 "2023-06-05T12:15:55Z")
Hi @nielsr
I was experimenting with the same notebook. In my case, I have trained on a custom dataset. Now, I want it to predict masks automatically (without any prompts like bounding boxes). How to do it?
I tried a few things (`input_boxes=None`, `input_boxes = [0,0,img_width, img_height]`, using `pipeline("mask-generator", model=my_model, processor = my_processor)` from [here](https://github.com/huggingface/notebooks/blob/main/examples/automatic_mask_generation.ipynb) and loading my checkpoint like in [here](https://github.com/facebookresearch/segment-anything/blob/main/notebooks/automatic_mask_generator_example.ipynb)) but they didn’t work.
---
**Author:**  [@marcomameli01](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmarcomameli01)%5C
**Post date:** [September 5, 2023, 6:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsam-image-size-for-fine-tuning%2F39666%2F4 "2023-09-05T18:27:20Z")
To use the model finetuned with the pipeline you can save the processor and tuned model with save\_pretrained and pass the same folder, after that you can pass to the pipeline the path where you save the with the save\_pretrained method.
---
**Author:**  [@rwood-97](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frwood-97)%5C
**Post date:** [November 8, 2023, 12:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsam-image-size-for-fine-tuning%2F39666%2F5 "2023-11-08T12:09:10Z")
>
>
> \### Feature request
>
> The \[SAM image processor\](https://github.com/huggingface/tr…ansformers/blob/main/src/transformers/models/sam/image\_processing\_sam.py) takes images as input and resizes them so that the longest edge is 1024 (using default values). This is the size expect as input fo the SAM model.
> For inference, this works fine as only the images need resizing but for fine-tuning as per \[this tutorial\](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SAM/Fine\_tune\_SAM\_(segment\_anything)\_on\_a\_custom\_dataset.ipynb), you need to resize both your images and your masks as the SAM model produces \`pred\_masks\` with size 256x256. If I don't resize my masks I get \`ground truth has different shape (torch.Size(\[2, 1, 768, 1024\])) from input (torch.Size(\[2, 1, 256, 256\]))\` when trying to calculate loss.
>
> To fix this, I've currently written a resize and pad function into my code:
>
> \`\`\`
> from PIL import Image
>
> def resize\_mask(image):
> longest\_edge = 256
>
> # get new size
> w, h = image.size
> scale = longest\_edge \* 1.0 / max(h, w)
> new\_h, new\_w = h \* scale, w \* scale
> new\_h = int(new\_h + 0.5)
> new\_w = int(new\_w + 0.5)
>
> resized\_image = image.resize((new\_w, new\_h), resample=Image.Resampling.BILINEAR)
> return resized\_image
>
> def pad\_mask(image):
> pad\_height = 256 - image.height
> pad\_width = 256 - image.width
>
> padding = ((0, pad\_height), (0, pad\_width))
> padded\_image = np.pad(image, padding, mode="constant")
> return padded\_image
>
> def process\_mask(image):
> resized\_mask = resize\_mask(image)
> padded\_mask = pad\_mask(resized\_mask)
> return padded\_mask
> \`\`\`
>
> and then have added this to my definition of SAMDataset:
>
> \`\`\`
> class SAMDataset(Dataset):
> def \_\_init\_\_(self, dataset, processor, transform = None):
> self.dataset = dataset
> self.processor = processor
> self.transform = transform
>
> def \_\_len\_\_(self):
> return len(self.dataset)
>
> def \_\_getitem\_\_(self, idx):
> item = self.dataset\[idx\]
>
> if self.transform:
> image = self.transform(item\["pixel\_values"\])
> else:
> image = item\["pixel\_values"\]
>
> # get bounding box prompt
> padded\_mask = process\_mask(item\["label"\])
> prompt = get\_bounding\_box(padded\_mask)
>
> # prepare image and prompt for the model
> inputs = self.processor(image, input\_boxes=\[\[prompt\]\], return\_tensors="pt")
>
> # remove batch dimension which the processor adds by default
> inputs = {k:v.squeeze(0) for k,v in inputs.items()}
>
> # add ground truth segmentation
> inputs\["ground\_truth\_mask"\] = padded\_mask
>
> return inputs
> \`\`\`
>
> This seems to work fine.
>
> What I think would be good is to allow input of masks in the SAM image processor. For example, the \[Segformer image processor\](https://github.com/huggingface/transformers/blob/v4.35.0/src/transformers/models/segformer/image\_processing\_segformer.py#L305) takes images and masks as inputs and resizes both to the size expected by the Segformer model.
>
> I have also seen there is a 'post\_process\_mask' method in the SAM image processor but I am unsure how to implement this in the tutorial I'm following. If you think this is a better way vs. what I am suggesting then please could you explain where I would add this in the code from the tutorial notebook.
>
> \### Motivation
>
> Easier fine tuning of SAM model.
>
> \### Your contribution
>
> I could try write a PR for this and/or make a PR to update the \[notebook\](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SAM/Fine\_tune\_SAM\_(segment\_anything)\_on\_a\_custom\_dataset.ipynb) instead .
Hi, I’ve just made an issue about this as had the same problem. My inputs + masks are not all the same size so couldn’t work out how to implement the solution you suggested but just resized my masks instead. Just thought would be good to comment here incase others find this thread.
---
**Author:**  [@Ziye-Thomas](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FZiye-Thomas)%5C
**Post date:** [April 3, 2024, 6:36am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsam-image-size-for-fine-tuning%2F39666%2F6 "2024-04-03T06:36:47Z")
Hi, nielsr! I tried the method you mentioned in my case, to resize the prediction mask (256 x 256) to the same shape as the ground truth (1080 x 1920), but there are still problems. The most intuitive issue is that when I visualize the resized mask, I can see that the position of mask has shifted significantly compared to the ground truth. However, this problem does not occur with masks post-processed using processor.image\_processor.post\_process\_masks. Do you have any suggestions for this issue? Or could you share what specific method is used in post\_process\_mask?