# Custom dataset maskformer
**URL:**
**Author:**  [@Mohamedfarag96](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMohamedfarag96)%5C
**Post date:** [January 16, 2025, 12:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-maskformer%2F136079%2F1 "2025-01-16T12:56:34Z")
Hello everyone,
wish you to be in good health!
I am facing an issue for training the maskformer on custom dataset.
my data has 6 classes, it is agricultural dataset. For pepper
I process the data as follows, firstly I created the masks using these lines
```auto
def __getitem__ (self, idx):
# Get image info
image_info = self.images[idx]
image_id = image_info['id']
width, height = image_info['width'], image_info['height']
# Load the image
relative_path = image_info['path'].lstrip('/datasets/') # Remove '/datasets/' prefix
image_path = os.path.join(self.root_dir, relative_path)
image = Image.open(image_path).convert("RGB")
if self.transform:
image = self.transform(image)
# Get annotations for the image
annotations = self.image_id_to_annotations.get(image_id, [])
segmentations = [anno.get('segmentation', []) for anno in annotations]
category_ids = [anno['category_id'] for anno in annotations]
# Create the semantic map
semantic_map = np.zeros((height, width), dtype=np.uint8)
for seg, category_id in zip(segmentations, category_ids):
if category_id == 11: # Skip category 11
continue
for polygon in seg:
polygon = np.array(polygon).reshape(-1, 2)
rr, cc = skimage.draw.polygon(polygon[:, 1], polygon[:, 0], semantic_map.shape)
semantic_map[rr, cc] = category_id
semantic_map_tensor = torch.tensor(semantic_map, dtype=torch.uint8)
return {
'image': image,
'semantic_map': semantic_map_tensor,
'image_id': image_id,
'width': width,
'height': height,
}
```
then to do the id2label I used this
```auto
"Needs to be checked later, date 13/01/2025, line: 193 new, and line 194 old"
# id2label = {10:"background", 11: "pepper_kp", 12: "pepper red", 13: "pepper yellow", 14: "pepper green", 15: "pepper mixed", 17: "pepper mixed_red", 18: "pepper mixed_yellow"}
id2label = {11:'background', 12: "pepper red", 13: "pepper yellow", 14: "pepper green", 15: "pepper mixed", 17: "pepper mixed_red", 18: "pepper mixed_yellow"}
# Remap to contiguous IDs
label2id = {old_id: new_id for new_id, old_id in enumerate(sorted(id2label.keys()))}
id2label_remapped = {label2id[old_id]: label for old_id, label in id2label.items()}
```
```auto
id2color = {
0: '#000000',
1: '#c7211c',
2: '#fff700',
3: '#00ff00',
4: '#e100ff',
5: '#ff6600',
6: '#d1c415',
}
palette = [
tuple(int(id2color[id].lstrip('#')[i:i+2], 16) for i in (0, 2, 4)) if id in id2color else (0, 0, 0)
for id in range(len(id2label_remapped))
]
palette = np.array(palette, dtype=np.uint8)
print(palette)
```
```auto
from transformers import MaskFormerImageProcessor
# Create a preprocessor
preprocessor = MaskFormerImageProcessor(ignore_index = 0, reduce_labels=False, do_resize=False, do_rescale=False, do_normalize=False)
```
is this the correct way to preprocess the inputs before fine tuning?
---
**Author:**  [@Mohamedfarag96](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMohamedfarag96)%5C
**Post date:** [January 16, 2025, 12:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-maskformer%2F136079%2F2 "2025-01-16T12:57:22Z")
```auto
model = MaskFormerForInstanceSegmentation.from_pretrained("facebook/maskformer-swin-base-ade",
id2label=id2label_remapped,
ignore_mismatched_sizes=True)
```
---
**Author:**  [@olmobaldoni](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Folmobaldoni)%5C
**Post date:** [January 16, 2025, 3:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-maskformer%2F136079%2F3 "2025-01-16T15:58:14Z")
Hi, I too am trying to finetune maskformer for a very similar task and am having problems with the parameter setting. In my case {0: ‘background’, 1: ‘unhealty’} and I am using:
```
self.processor = AutoImageProcessor.from_pretrained(
‘facebook/maskformer-swin-small-coco’,
do_reduce_labels=True,
ignore_index=255,
do_resize=False,
do_resize=False,
do_normalize=False,
)
```
because I do not want to predict the background (if I understand correctly). But this way the results are really very poor and the model does not seem to learn anything.
The only way I got good results was to consider the background, but in that case the model gives too much weight to the background class and does not predict everything else correctly.
The strange thing is that using mask2former on the same dataset and with the same id2label I get far better results
---
**Author:**  [@Mohamedfarag96](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMohamedfarag96)%5C
**Post date:** [January 16, 2025, 5:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-maskformer%2F136079%2F4 "2025-01-16T17:01:47Z")
Hey olmo!
Thanks a lot!
I am trying also to do everything to get good results. furthermore, I want to add class weights for each class, I have severe imbalance unfortunately. But it is not that easy.
also I have weird colored background, I don’t know why. I am trying to ignore the background but not helping so much unfortunately
---
**Author:**  [@Mohamedfarag96](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMohamedfarag96)%5C
**Post date:** [January 16, 2025, 5:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-maskformer%2F136079%2F5 "2025-01-16T17:13:18Z")
but why do use reduce labels? I am doing the same but with just ignoring index=0
---
**Author:**  [@olmobaldoni](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Folmobaldoni)%5C
**Post date:** [January 16, 2025, 7:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-maskformer%2F136079%2F6 "2025-01-16T19:43:29Z")
* * *
I decided to use `do_reduce_labels=True` together with `ignore_index=255`, following this discussion about a similar case:
> [@Understanding ignore\_index and reduce\_labels](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funderstanding-ignore-index-and-reduce-labels%2F64587%2F2)%3A
>
> Ignore\_index is the index that is ignored for loss calculations etc. set to 255 - as that’s used in standard loss functions I believe. Some models default to 255 and others don’t set it - not sure exactly why. This will take your background pixels and also unlabelled pixels and set them to the ignore\_index value Reduce\_labels - set to True. reindexes the label values so that they start from 0 after the background is set to 255
Additionally, I found a similar scenario in a tutorial for fine-tuning Mask2Former, where the `config.json` file also has `do_reduce_labels=True`. According to the documentation, the preprocessing for MaskFormer and Mask2Former should be identical.
>
In the same file:
```python
# We need to specify the label2id mapping for the model
# it is a mapping from semantic class name to class index.
# In case your dataset does not provide it, you can create it manually:
# label2id = {"background": 0, "cat": 1, "dog": 2}
label2id = dataset["train"][0]["semantic_class_to_id"]
if args.do_reduce_labels:
label2id = {name: idx for name, idx in label2id.items() if idx != 0} # remove background class
label2id = {name: idx - 1 for name, idx in label2id.items()} # shift class indices by -1
```
From what I gather (though I’m not entirely sure, as the documentation isn’t very clear), when you don’t want to consider the background as a segmentable class, the preprocessor replaces the background in the image with the value `255`. This value is ignored during loss computation.
Thus, I set the parameter as follows. I don’t think the `ignore_index` value can be arbitrarily set (e.g., if I have `{0: 'garden', 1: 'car', 2: 'tree'}` and set `ignore_index=1`, the ‘car’ class will be ignored during loss computation).
The parameter `do_reduce_labels=True` ensures that classes start from 0 and increment upward, which is why they are shifted by `-1`.
* * *
Example (Models trained with 20 epochs and learning rate `5e-5`)
Test Image:
* * *
Preprocessor for MaskFormer:
```python
self.processor = AutoImageProcessor.from_pretrained(
"facebook/maskformer-swin-small-coco",
do_reduce_labels=True,
reduce_labels=True,
ignore_index=255,
do_resize=False,
do_rescale=False,
do_normalize=False,
)
```
Results with MaskFormer:
```plaintext
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Test metric DataLoader 0
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
test_loss 1.0081120729446411
test_map 0.038004860281944275
test_map_50 0.06367719173431396
test_map_75 0.040859635919332504
test_map_large 0.5004204511642456
test_map_medium 0.04175732284784317
test_map_small 0.007470746990293264
test_mar_1 0.01011560671031475
test_mar_10 0.05838150158524513
test_mar_100 0.06329479813575745
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
```
Test Image Result with MaskFormer:
* * *
Preprocessor for Mask2Former:
```python
self.id2label = {0: "unhealty"}
self.label2id = {v: int(k) for k, v in self.id2label.items()}
self.processor = AutoImageProcessor.from_pretrained(
"facebook/mask2former-swin-small-coco-instance",
do_reduce_labels=True,
reduce_labels=True,
ignore_index=255,
do_resize=False,
do_rescale=False,
do_normalize=False,
)
```
Results with Mask2Former:
```plaintext
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Test metric DataLoader 0
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
test_loss 15.374979972839355
test_map 0.44928184151649475
test_map_50 0.6224347949028015
test_map_75 0.5011898279190063
test_map_large 0.8390558958053589
test_map_medium 0.6270320415496826
test_map_small 0.32075226306915283
test_mar_1 0.03526011481881142
test_mar_10 0.24104046821594238
test_mar_100 0.5274566411972046
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
```
Test Image Result with Mask2Former:
* * *
As you can see, the results are very different, even though the code is identical, except for the parts where the model type is changed. if you want I can share the code.
---
**Author:**  [@Mohamedfarag96](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMohamedfarag96)%5C
**Post date:** [January 17, 2025, 10:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-maskformer%2F136079%2F7 "2025-01-17T10:25:00Z")
Thaaaaaanks for the details!
I will check and try if it works for me!
you are right, the documentation isn’t clear and the tutorials is not that generic to be adapted to any case.
for my case the background is with index or label 0, hence I am ignoring index 0 to drop it from calculations during fine tuning.
but without reducing labels.
Two main issues for me now, the plotting but maybe I need to figure out a solution for that.
and the fine tuning procedure, which backbone should I use and which hyper-parameters values. I have tried swin base for ADE data and also ResNet-50, but no good solution till now.
also I have tried coco weights for swin-base, but not that good
this is the final result
---
**Author:**  [@Mohamedfarag96](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMohamedfarag96)%5C
**Post date:** [January 17, 2025, 10:43am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-maskformer%2F136079%2F8 "2025-01-17T10:43:10Z")
my problem now is that the model is predicting wrong class for the background, I don’t know why
---
**Author:**  [@Mohamedfarag96](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMohamedfarag96)%5C
**Post date:** [January 17, 2025, 11:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-maskformer%2F136079%2F9 "2025-01-17T11:11:29Z")
**Update** I have tried Mask2Former, and It is so much better by far.
---
**Author:**  [@Mohamedfarag96](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMohamedfarag96)%5C
**Post date:** [January 17, 2025, 11:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-maskformer%2F136079%2F10 "2025-01-17T11:13:37Z")
Mask2Former outputs!
---
**Author:**  [@Mohamedfarag96](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMohamedfarag96)%5C
**Post date:** [January 17, 2025, 12:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-maskformer%2F136079%2F11 "2025-01-17T12:00:25Z")
also I have problem, I have a dominant class, and I need to add class weight to the loss to balance the classes, do you know by any means how to do it?
---
**Author:**  [@Mohamedfarag96](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMohamedfarag96)%5C
**Post date:** [January 17, 2025, 3:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-maskformer%2F136079%2F12 "2025-01-17T15:23:02Z")
Dear Olmo! Thaaaaaaanks a lot for your input it solved my problem, by setting the ignore index to 255. ❤
I am speechless! wish you a nice weekend! and good luck with your work, you do impressive things from what I see from the images!
Here are some images! wish you a nice day!
---
**Author:**  [@olmobaldoni](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Folmobaldoni)%5C
**Post date:** [January 17, 2025, 5:12pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-maskformer%2F136079%2F13 "2025-01-17T17:12:32Z")
Hi Mohamed! I’m very happy that you managed to solve the problem! May I just ask you if you were able to resolve the issue with MaskFormer or with Mask2Former? I would also like to ask if you could send the processor you used, along with the id2label mapping. I’m curious to know how you handle the background class (from the image, it seems that you still have the background class).
I wish you a good weekend as well!
---
**Author:**  [@Mohamedfarag96](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMohamedfarag96)%5C
**Post date:** [January 17, 2025, 5:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-maskformer%2F136079%2F14 "2025-01-17T17:17:34Z")
I switched to Mask2Former, but next week I will check if it works for MaskFormer. They need to document in a better way 😆
Sure, Next week, Monday I will send you everything!
Even we can have a quick meeting if you want through zoom to talk about it.
Thanks a lot for your input!
---
**Author:**  [@olmobaldoni](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Folmobaldoni)%5C
**Post date:** [January 17, 2025, 5:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-maskformer%2F136079%2F15 "2025-01-17T17:27:55Z")
Sorry to bother you, but I was curious to understand why I have this drop in performance when I use MaskFormer instead of Mask2Former. Please send me everything only if it doesn’t bother you and doesn’t take up your time. If I eventually can’t understand why it’s not working, we can also talk on Zoom!
Thanks again for your kindness! 😁
---
**Author:**  [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [January 18, 2025, 5:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-maskformer%2F136079%2F16 "2025-01-18T05:28:20Z")
Closed on January 18, 2025, 5:28am UTC
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.