# How to load large-scale text-image pair dataset

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-large-scale-text-image-pair-dataset%2F52939%3E%5C
**Category:** 🤗Datasets\
**Created:** [August 31, 2023, 8:21am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-large-scale-text-image-pair-dataset%2F52939 "2023-08-31T08:21:05Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ywlee88](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ywlee88/32/19043_2.png) [@ywlee88](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fywlee88)%5C
**Post date:** [August 31, 2023, 8:21am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-large-scale-text-image-pair-dataset%2F52939%2F1 "2023-08-31T08:21:05Z")

</div>

Hi,

I have tried to finetune SDXL model on a subset of LAION-aesthetic-5+ (89M) using the [example code](https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_sdxl.py).

I used this code for loading data:

> dataset = load\_dataset(“imagefolder”, data\_dir=args.train\_data\_dir, split=“train”)

`args.train_data_dir` denote the data directory including over 89M image-text pairs.

But the data loading time is too long ☹ and eventually it failed to load the data.

For this large-scale Text-to-Image mode training, is there a more efficient way?

Thanks in advance 🙂

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [August 31, 2023, 6:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-large-scale-text-image-pair-dataset%2F52939%2F2 "2023-08-31T18:58:28Z")

</div>

ImageFolder’s file resolution is currently not optimized for large datasets like this one. In your case, it’s best to create a dataset loading script or use `Dataset.from_generator` (with a generator that yields `{"image": pil_image, "text": text}` dictionaries) instead of `load_dataset` to generate the dataset.

---

<div class="post-metadata">

**Author:** ![mikehemberger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mikehemberger/32/28013_2.png) [@mikehemberger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmikehemberger)%5C
**Post date:** [December 12, 2023, 1:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-large-scale-text-image-pair-dataset%2F52939%2F3 "2023-12-12T13:26:45Z")

</div>

Hi @ywlee88 ,  
Did the loading script solution work for you or did you find another way to load your data?

Best,  
Mike

---

<div class="post-metadata">

**Author:** ![panigrah](https://avatars.discourse-cdn.com/v4/letter/p/bb73d2/32.png) [@panigrah](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpanigrah)%5C
**Post date:** [December 13, 2023, 12:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-large-scale-text-image-pair-dataset%2F52939%2F4 "2023-12-13T00:28:32Z")

</div>

wouldn’t streaming work here? setting `streaming=True` on the load\_dataset call? My understanding is that as long as you are not trying to randomize the training data, streaming should work.

---

<div class="post-metadata">

**Author:** ![Ryoo72](https://avatars.discourse-cdn.com/v4/letter/r/3da27b/32.png) [@Ryoo72](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRyoo72)%5C
**Post date:** [February 7, 2025, 11:21am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-large-scale-text-image-pair-dataset%2F52939%2F5 "2025-02-07T11:21:31Z")

</div>

@mariosasko  
I’m also experiencing difficulties in handling a large-scale image dataset in the millions. I’m curious if using `Dataset.from_generator` method works as map data (specifically, I’m wondering about fast access speed and whether shuffling is possible). Additionally, I’m wondering which method would be better: storing images in TAR and calling them, or storing image paths as strings and loading them in the collate function.

Thank you for always putting in so much effort for the community 🙂