# batched I/O from disk when load\_dataset API is used?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbatched-i-o-from-disk-when-load-dataset-api-is-used%2F137391%3E%5C
**Category:** 🤗Datasets\
**Created:** [January 26, 2025, 12:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbatched-i-o-from-disk-when-load-dataset-api-is-used%2F137391 "2025-01-26T12:06:03Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![mmatak](https://avatars.discourse-cdn.com/v4/letter/m/5f8ce5/32.png) [@mmatak](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmmatak)%5C
**Post date:** [January 26, 2025, 12:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbatched-i-o-from-disk-when-load-dataset-api-is-used%2F137391%2F1 "2025-01-26T12:06:03Z")

</div>

Hi,

I just downloaded a ~1TB [HF dataset](https://huggingface.co/datasets/ll4ma-lab/grasp-dataset) onto disk using git clone and am training a model using load\_dataset API in streamed mode. The stream would read samples from the disk and populate a buffer of samples, which I’d use to construct a batch for the model.

During this training, the bottleneck is reading from the disk. I wonder if there is a batched way to load from the disk when stream=True option? This is different from yielding a batch while reading samples 1 by 1.

Thank you!

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [January 26, 2025, 1:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbatched-i-o-from-disk-when-load-dataset-api-is-used%2F137391%2F2 "2025-01-26T13:24:59Z")

</div>

Is this it? It’s probably a little different…

> [@Streaming dataset and cache](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-dataset-and-cache%2F32751)%3A
>
> Hello! I was wondering about the inner working of dataset streaming. Is there any form of caching of the samples that will be needed soon or are they downloaded and processed only when the model requires them for training ? In the later case, we could expect a higher latency when going from a batch to another, especially when the samples require some heavy processing. In this case it’s probably better to process the dataset locally beforehand. Cheers

---

<div class="post-metadata">

**Author:** ![mmatak](https://avatars.discourse-cdn.com/v4/letter/m/5f8ce5/32.png) [@mmatak](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmmatak)%5C
**Post date:** [January 27, 2025, 6:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbatched-i-o-from-disk-when-load-dataset-api-is-used%2F137391%2F3 "2025-01-27T18:38:17Z")

</div>

That’s not quite it, but I guess I can look into source code to understand how things are loaded exactly. I think it’s fine to close the topic.