# Too many open files on big datasets

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftoo-many-open-files-on-big-datasets%2F109084%3E%5C
**Category:** 🤗Datasets\
**Created:** [September 27, 2024, 11:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftoo-many-open-files-on-big-datasets%2F109084 "2024-09-27T23:22:27Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [September 30, 2024, 10:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftoo-many-open-files-on-big-datasets%2F109084%2F4 "2024-09-30T22:45:43Z")

</div>

I see. Even in that case, there seems to be easier way. Not sure if it would work…

> [@How to load a large hf dataset efficiently?](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-large-hf-dataset-efficiently%2F69288%2F5)%3A
>
> @lhoestq what do you think about loading this dataset in a lazy way, for example loading it as a batch of 64 samples at a time? Is it more inefficient than loading those shards of data?

> **[Loading methods](https://huggingface.co/docs/datasets/package_reference/loading_methods#datasets.load_from_disk)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

```py
ds = load_from_disk('path/to/dataset/directory', keep_in_memory=True) # RAM consumption looks terrible... But won't disk access decrease?

```

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftoo-many-open-files-on-big-datasets%2F109084)._