# 5M small images (~100Gb)

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2F5m-small-images-100gb%2F19761%3E%5C
**Category:** 🤗Datasets\
**Created:** [June 29, 2022, 6:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2F5m-small-images-100gb%2F19761 "2022-06-29T06:52:59Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![yiwc](https://avatars.discourse-cdn.com/v4/letter/y/c67d28/32.png) [@yiwc](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fyiwc)%5C
**Post date:** [June 29, 2022, 6:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2F5m-small-images-100gb%2F19761%2F1 "2022-06-29T06:52:59Z")

</div>

I 'm collecting and training on a dataset of 5M small image files, which cost about 100Gb, what’s your recommended solution for this size dataset?

I’m trying imagefolder, looks super slow.

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [June 30, 2022, 3:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2F5m-small-images-100gb%2F19761%2F2 "2022-06-30T15:26:13Z")

</div>

Hi! Can you run the command `datasets-cli env` and paste the output here? Does passing `ignore_verifications=True` to `load_dataset` improve the loading speed? You can also interrupt the process while waiting for it to finish and paste the stack trace here to make it easier for us to debug this issue.

---

<div class="post-metadata">

**Author:** ![yiwc](https://avatars.discourse-cdn.com/v4/letter/y/c67d28/32.png) [@yiwc](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fyiwc)%5C
**Post date:** [July 5, 2022, 8:02am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2F5m-small-images-100gb%2F19761%2F3 "2022-07-05T08:02:39Z")

</div>

thank for response, I use load from JSON file, (the meta json), and process images using map with multiple workers, it works fine now.