# Too many open files on big datasets
**URL:**
**Author:**  [@Aceticia](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAceticia)%5C
**Post date:** [September 27, 2024, 11:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftoo-many-open-files-on-big-datasets%2F109084%2F1 "2024-09-27T23:22:27Z")
Hi all, I have a pretty large dataset which I split into many partitions, with each partition consisting of about 200 of 1G files using datasets. An issue I run into when loading is that when I try to load\_from\_disk all these different datasets together, I get “too many open files” error. I can’t increase my file limit.
My question is: Other than rerunning my entire dataset and storing them using bigger chunks like 50G (which will take a long time), what could be a good solution to solve this problem?
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [September 29, 2024, 7:14am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftoo-many-open-files-on-big-datasets%2F109084%2F2 "2024-09-29T07:14:23Z")
It seems to be a Linux-specific problem. I’m a Windows user, so I’m not sure, but there seem to be a couple of workarounds.
> **[How to Solve the "Too Many Open Files" Error on Linux](https://www.howtogeek.com/805629/too-many-open-files-linux/)**
>
> You need to get a handle on it.
---
**Author:**  [@Aceticia](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAceticia)%5C
**Post date:** [September 30, 2024, 8:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftoo-many-open-files-on-big-datasets%2F109084%2F3 "2024-09-30T20:33:11Z")
This is helpful, but unfortunately I can’t use the solutions there since I don’t have root access. After much thought, I guess the only two solutions are:
1. Remake my dataset
2. Load random subsets of my dataset for each epoch
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [September 30, 2024, 10:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftoo-many-open-files-on-big-datasets%2F109084%2F4 "2024-09-30T22:45:43Z")
I see. Even in that case, there seems to be easier way. Not sure if it would work…
> [@How to load a large hf dataset efficiently?](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-large-hf-dataset-efficiently%2F69288%2F5)%3A
>
> @lhoestq what do you think about loading this dataset in a lazy way, for example loading it as a batch of 64 samples at a time? Is it more inefficient than loading those shards of data?
> **[Loading methods](https://huggingface.co/docs/datasets/package_reference/loading_methods#datasets.load_from_disk)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
```py
ds = load_from_disk('path/to/dataset/directory', keep_in_memory=True) # RAM consumption looks terrible... But won't disk access decrease?
```