# Unable to load images
**URL:**
**Author:**  [@gksriharsha](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgksriharsha)%5C
**Post date:** [December 29, 2024, 3:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-load-images%2F133408%2F1 "2024-12-29T03:25:47Z")
Hello all!
I am trying to build an OCR image and have saved some image - text pairs in a dataset and uploaded them into huggingface repo. While fetching the same repo to another machine, it shows that some images are corrupted with the PIL.UnidentifiedimageError . I have
- Using the SQL checked if the image is really corrupted. It isn’t.
- Just to be on the safe side removed (12 instances which are shown to be “corrupted”) and reuploaded them after removing the rows. Now 12 different rows are being shown as “corrupted” . I have also checked these ones too . They are being perfectly rendered on the website.
To try to mitigate this issue, I have updated the datasets version to the latest (3.2.0) on both the uploading system and the compute system that is downloading the data. Unfortunately this is where I am stuck.
Is anyone else facing any similar issue ?
Thanks a lot.
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [December 29, 2024, 6:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-load-images%2F133408%2F2 "2024-12-29T06:25:03Z")
I’ve never heard of that error before, but the behavior seems like a bug…
It might be quicker to raise an issue on github.
> **[Issues · huggingface/datasets](https://github.com/huggingface/datasets/issues)**
>
> 🤗 The largest hub of ready-to-use datasets for ML models with fast, easy-to-use and efficient data manipulation tools - Issues · huggingface/datasets
Also, if the image data set is extremely large, there seems to be a trick to how to create it.
>
>
> \### Describe the bug
>
> While testing image dataset creation, I'm seeing significa…nt performance bottlenecks with imagefolders when scanning a directory structure with large number of images.
>
>
> \## Setup
> \* Nested directories (5 levels deep)
> \* 3M+ images
> \* 1 \`metadata.jsonl\` file
>
>
> \## Performance Degradation Point 1
>
> Degradation occurs because \[\`get\_data\_files\_patterns\`\](https://github.com/huggingface/datasets/blob/main/src/datasets/data\_files.py#L231-L243) runs the exact same scan for many different types of patterns, and there doesn't seem to be a way to easily limit this. It's controlled by the definition of \[\`ALL\_DEFAULT\_PATTERNS\`\](https://github.com/huggingface/datasets/blob/main/src/datasets/data\_files.py#L82-L85).
>
> One scan with 3M+ files takes about 10-15 minutes to complete on my setup, so having those extra scans really slows things down – from 10 minutes to 60+. Most of the scans return no matches, but they still take a significant amount of time to complete – hence the poor performance.
>
> As a side effect, when this scan is run on 3M+ image files, Python also consumes up to 12 GB of RAM, which is not ideal.
>
>
> \## Performance Degradation Point 2
>
> The second performance bottleneck is in \[\`PackagedDatasetModuleFactory.get\_module\`\](https://github.com/huggingface/datasets/blob/d7dfbc83d68e87ba002c5eb2555f7a932e59038a/src/datasets/load.py#L707-L711), which calls \`DataFilesDict.from\_local\_or\_remote\`.
>
> It runs for a long time (60min+), consuming significant amounts of RAM – even more than the point 1 above. Based on \`iostat -d 2\`, it performs \*\*zero\*\* disk operations, which to me suggests that there is a code based bottleneck there that could be sorted out.
>
> \### Steps to reproduce the bug
>
> \`\`\`python
> from datasets import load\_dataset
> import os
> import huggingface\_hub
>
> dataset = load\_dataset(
> 'imagefolder',
> data\_dir='/some/path',
> # just to spell it out:
> split=None,
> drop\_labels=True,
> keep\_in\_memory=False
> )
>
> dataset.push\_to\_hub('account/dataset', private=True)
> \`\`\`
>
> \### Expected behavior
>
> While it's certainly possible to write a custom loader to replace \`ImageFolder\` with, it'd be great if the off-the-shelf \`ImageFolder\` would by default have a setup that can scale to large datasets.
>
> Or perhaps there could be a dedicated loader just for large datasets that trades off flexibility for performance? As in, maybe you have to define explicitly how you want it to work rather than it trying to guess your data structure like \`\_get\_data\_files\_patterns()\` does?
>
> \### Environment info
>
> \- \`datasets\` version: 2.7.1
> \- Platform: Linux-4.14.296-222.539.amzn2.x86\_64-x86\_64-with-glibc2.2.5
> \- Python version: 3.7.10
> \- PyArrow version: 10.0.1
> \- Pandas version: 1.3.5
---
**Author:**  [@gksriharsha](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgksriharsha)%5C
**Post date:** [December 31, 2024, 5:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-load-images%2F133408%2F3 "2024-12-31T17:32:14Z")
Thank you for pointing me to datasets github repo. After a lot of debugging of many 100s of images in the dataset, I found that the bug is actually in my augmentation part of the code. The issue is with not accounting for race conditions.