# Performance Issue Segformer Cityscapes 1024-1024

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fperformance-issue-segformer-cityscapes-1024-1024%2F45584%3E%5C
**Category:** Models\
**Created:** [July 4, 2023, 8:49am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fperformance-issue-segformer-cityscapes-1024-1024%2F45584 "2023-07-04T08:49:52Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![hporta](https://avatars.discourse-cdn.com/v4/letter/h/898d66/32.png) [@hporta](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhporta)%5C
**Post date:** [July 4, 2023, 8:49am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fperformance-issue-segformer-cityscapes-1024-1024%2F45584%2F1 "2023-07-04T08:49:52Z")

</div>

First, thanks for sharing the weights of Segformer on CityScapes [here](https://huggingface.co/nvidia/segformer-b0-finetuned-cityscapes-1024-1024). I have difficulties replicating (or even approaching) the Segformer paper performance on cityscapes. The first issue I observed is that from B0 to B5 Image Processors only B1 and B5 resize the input image to 1024_1024 and the rest to 512_ 512, as you can see in the attached screenshot.

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/1/9/190d3c38c8282e92d1a71b48f2687b3dd067bc14.jpeg)

Moreover, with a standard pytorch Dataset (see below using the LabelTrainingIds after running cityscapesscripts) loading the image and label as PIL image and passing both through the image processor I obtain an mIoU of 58 on cityscapes validation.

When using a custom albumentation transform pipeline of Resize(1024, 1024) , Normalize, ToTensorV2 I improve the result to mean IoU 68 still far from the 76.2 in the paper.

From the paper and mmsegmentation the test was done via 1024\*1024 sliding window with stride 768. Were the result replicated here with this type of inference, it is unclear from the image processor attributes ? Is there an available implementation of the Cityscapes inference pipeline with this model implementation ?

If not what was the achieved results and the used pre-processing pipeline ?

Code for CityScapes Dataloader:

```auto
class CityscapesDataset(Dataset):
    """CityScapes Dataset from Raw Data"""

    def __init__ (
        self,
        root_dir: os.PathLike,
        image_processor: Optional[BaseImageProcessor] = None,
        transform: Optional[A.Compose] = None,
        split: str = "train",
    ): # TODO: we could specify a callable for transform
        """Initialize the dataset Object.

        Args:
            root_dir (os.PathLike): Local path to raw data.
            image_processor (Optional[BaseImageProcessor], optional): HuggingFace Image Processor. Defaults to None.
            transform (Optional[A.Compose], optional): Set of transforms for processing images and masks. Defaults to None.
            split (str, optional): Dataset split to load. Defaults to "train".
        """
        self.root_dir = root_dir
        self.image_processor = image_processor
        self.split = split
        self.transform = transform

        self.images_dir = self.root_dir / "leftImg8bit" / self.split
        self.labels_dir = self.root_dir / "gtFine" / self.split

        self.images = []
        self.labels = []
        self.ids = []

        for city in self.images_dir.iterdir():
            for image_path in city.iterdir():
                self.images.append(image_path)
                self.labels.append(
                    self.labels_dir / city.name / image_path.name.replace("leftImg8bit", "gtFine_labelTrainIds")
                )
                self.ids.append(image_path.name.replace("_leftImg8bit.png", ""))

    def __len__ (self) -> int:
        return len(self.images)

    def __getitem__ (self, idx) -> Dict[str, torch.Tensor]:
        image = Image.open(self.images[idx]).convert("RGB")
        label = Image.open(self.labels[idx])

        if self.transform is not None:
            transformed = self.transform(image=np.asarray(image).copy(), mask=np.asarray(label).copy())
            image = transformed["image"]
            label = transformed["mask"]

        if self.image_processor is not None:
            encoded_inputs = {}
            process_inputs = self.image_processor.preprocess(images=image, segmentation_maps=label, return_tensors="pt")
            for k, _ in process_inputs.items():
                encoded_inputs[k] = process_inputs[k].squeeze() # remove batch dimension

        else:
            encoded_inputs = {"pixel_values": image, "labels": label}

        encoded_inputs["id"] = self.ids[idx]
        # encoded_inputs["labels"].apply_(lambda x: CityscapesDataset.mapping_ids[x])

        return encoded_inputs

```