# Encoding masks for Mask2Former and Panopic Segmentation
**URL:**
**Author:**  [@nlassaux](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnlassaux)%5C
**Post date:** [October 5, 2024, 2:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fencoding-masks-for-mask2former-and-panopic-segmentation%2F110245%2F1 "2024-10-05T14:36:17Z")
Hi everyone!
I’m trying to set up a fine-tuning for a panopic segmentation task. I managed to get everything working for semantic segmentations, with the mask being the category id for each polygon. It works with a single channel.
Now, I’d like to “upgrade” it to panoptic, so I need to encode the instance\_id in the mask. I saw some tutorials using RGB, with the first channel being the category and the second an instance id, others using a single channel and byte operations to store both together.
I guess my general question is: what format does the model expect to see as input as label mask?
If you have any insight you could share, thank you very much! 🙂
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [October 6, 2024, 11:08am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fencoding-masks-for-mask2former-and-panopic-segmentation%2F110245%2F2 "2024-10-06T11:08:03Z")
I have no idea what the essential parts are, but it is possible that this article could tell us just the options to pass on.
> **[Universal Image Segmentation with Mask2Former and OneFormer](https://huggingface.co/blog/mask2former)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
> **[Image Segmentation](https://huggingface.co/docs/transformers/tasks/semantic_segmentation)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
---
**Author:**  [@nlassaux](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnlassaux)%5C
**Post date:** [October 9, 2024, 8:59am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fencoding-masks-for-mask2former-and-panopic-segmentation%2F110245%2F3 "2024-10-09T08:59:01Z")
In case it helps everyone, here is the solution I found:
```auto
panoptic_seg_gt = rgb_to_id(panoptic_seg_gt)
inputs = self.processor(
[image],
[panoptic_seg_gt],
instance_id_to_semantic_id=inst2class,
return_tensors="pt",
)
```
- image: is a channel-first RGB picture (call `image.transpose(2, 0, 1)` on it)
- panoptic\_seg\_gt: is a RGB picture that maps the item\_id into RGB space using the `rgb_to_id` function. That was the main point I misunderstood: we don’t need to pass the category\_id here, because that information is contained in the dictionary we pass into `instance_id_to_semantic_id` and the processor automatically maps item\_ids to category\_ids. Yes, item\_ids can get very large, but it’s encoded on the 3 channels, so it can store 16M+ different items.
- instance\_id\_to\_semantic\_id: is a dictionary that maps items to their categories.