# Support of very large dataset?
**URL:**
**Author:**  [@mralexis](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmralexis)%5C
**Post date:** [June 15, 2021, 11:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsupport-of-very-large-dataset%2F6872%2F1 "2021-06-15T23:03:58Z")
What would be the recommended usage of `datasets` given I have large dataset e.g. common crawl, and need distributed training? For example, is there a build-in functionality that I could preprocess the data once and save/load in disk in a binarized/efficient way? And is there anything worth noticing for efficient distributed training with large datasets?
Tried to go over the doc but didnāt find anything on this.
Thanks!
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [June 16, 2021, 2:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsupport-of-very-large-dataset%2F6872%2F2 "2021-06-16T14:14:57Z")
Hi ! Sure the `datasets` library is designed to support the processing of large scale datasets. Datasets are loaded using memory mapping from your disk so it doesnāt fill your RAM. You can parallelize your data processing using `map` since it supports multiprocessing. Then you can save your processed dataset using `save_to_disk`, and reload it later using `load_from_disk`
```python
from datasets import load_dataset, load_from_disk
dataset = load_dataset(...)
dataset = dataset.map(..., num_proc=num_processes)
dataset.save_to_disk("path/to/save/directory")
# later
dataset = load_from_disk("path/to/save/directory")
```
Things worth noticing:
- you can specify a `cache_dir` parameter in `load_dataset` so that you can store the raw + prepared data wherever you want and be able to delete them later to save space if needed.
- If you are working on a cluster with a virtual filesystem, you may want to make sure that the memory mapping works efficiently. This is probably the case if you are doing distributed training. There is a discussion about this in [here](https://github.com/huggingface/datasets/issues/2252) if this is the case. We are still investigating why some virtual filesystems have such behaviors.
---
**Author:**  [@mralexis](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmralexis)%5C
**Post date:** [June 16, 2021, 2:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsupport-of-very-large-dataset%2F6872%2F3 "2021-06-16T14:46:14Z")
Thanks for the great reply! This is very helpful.
Quick questions:
1. Should I do data preprocessing before launching a distributed training job? In other words, would the ideal case be just having `dataset = load_from_disk("path/to/save/directory")` in distributed training script?
2. Is there any built-in prefetch function like the one in e.g. fairseq? Or is this already taken care of?
3. Thanks for the great reply! This is very helpful.
Quick questions:
1. Should I do data preprocessing before launching a distributed training job? In other words, would the ideal case be just having `dataset = load_from_disk("path/to/save/directory")` in distributed training script?
2. Is there any built-in prefetch function like the one in e.g. fairseq? Or is this already taken care of?
3. Out of curiosity, how was the shuffling done if not all data is loaded into RAM?
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [June 16, 2021, 3:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsupport-of-very-large-dataset%2F6872%2F4 "2021-06-16T15:07:03Z")
1. Sure this is the easiest way to load your processed dataset in your training script
2. When you load the dataset, then the full dataset is loaded from your disk. Thereās no prefetch function: you can directly access any element at any position in your dataset.
3. The shuffling is done by shuffling the index of the dataset (i.e. the mapping between what ` __getitem__ ` returns and the actual position of the examples on disk). The actual elements on disk are not shuffled. Shuggling the index is done in memory though, and the resulting index is written to disk and loaded from the disk via memory mapping afterwards.
---
**Author:**  [@mralexis](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmralexis)%5C
**Post date:** [June 16, 2021, 3:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsupport-of-very-large-dataset%2F6872%2F5 "2021-06-16T15:16:47Z")
Thanks!
For 2) would this be the bottomneck if there is no prefetch?
For 3) would this be fast enough? I guess it would be costly to get an arbitrary index from disk?
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [June 16, 2021, 4:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsupport-of-very-large-dataset%2F6872%2F6 "2021-06-16T16:31:32Z")
1. The bottleneck is in general the I/O limitations provided by memory mapping, which depends on your hardware.
2. Get examples from an arbitrary positions from disk is pretty fast - we used the arrow format especially for that. Itās in general a matter of milliseconds. Shuffling the index on the other hand might be pretty slow if you have hundreds of billions of examples since it tries to do exact shuffling. This can be improved by using an approximate shuffling method (though we currently donāt have one in the library)
Hope that helps š
---
**Author:**  [@mralexis](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmralexis)%5C
**Post date:** [June 16, 2021, 4:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsupport-of-very-large-dataset%2F6872%2F7 "2021-06-16T16:56:41Z")
Thanks a lot, Quentin!
---
**Author:**  [@jasonyoun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjasonyoun)%5C
**Post date:** [June 26, 2021, 8:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsupport-of-very-large-dataset%2F6872%2F8 "2021-06-26T08:09:57Z")
I assume youāll be applying tokenizers or other processing to that large dataset using `.map()`? In that case, what was your `batch_size` and `writer_batch_size`? I found it very difficult to find the right numbers that do not consume all my RAM and was _fast enough_ (my dataset is \>100GB).
---
**Author:**  [@khel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkhel)%5C
**Post date:** [September 2, 2021, 2:21am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsupport-of-very-large-dataset%2F6872%2F9 "2021-09-02T02:21:04Z")
@lhoestq
Hi lhoestq! Thanks for explaninig how to handle very large dataset.
I have another question about `save_to_disk` and `load_from_disk`.
My dataset has a lot of files (#files : 10000) and its size is bigger than 5T.
The workflow involves preprocessing and saving its result using `save_to_disk` **per file** (or it takes a long time to make tables).
So it results 10000 arrow files. (total size : 8T)
After this, I edit `state.json` of one of directories that saved so that its `"_data_files"` points arrow files that I made.
The problem is that `load_from_disk` takes a lot of time (like 6 hours or 3 hours for reloading)
Could I reduce the time it takes? or Could you share another great way to preprocess files?
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [September 6, 2021, 1:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsupport-of-very-large-dataset%2F6872%2F10 "2021-09-06T13:08:53Z")
Hi ! Did you try using `load_from_disk` on each one of the 10,000 arrow files and then use `concatenate_datasets` to get the full dataset ?
---
**Author:**  [@khel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkhel)%5C
**Post date:** [September 7, 2021, 1:39am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsupport-of-very-large-dataset%2F6872%2F11 "2021-09-07T01:39:03Z")
Thanks for the suggestion! I figured out that [this](https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_dataset.py#L762-L768) part takes a lot of time. So, I just load each one of the files `using load_from_disk ` and concat them as you said.
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [September 14, 2021, 2:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsupport-of-very-large-dataset%2F6872%2F12 "2021-09-14T14:28:56Z")
Thanks for diving into it š
The URL you shared seems to point to a dynamic code snippet (from the master branch), that got some changes a few days ago. For future reference, the āpart that takes a lot of timeā is [here](https://github.com/huggingface/datasets/blob/f4ec065efdf4bb9db9103dccab768608fd40c6b3/src/datasets/arrow_dataset.py#L762-L768)
Do you know by any chance what part of the code exactly in the Dataset initialization caused some slowdowns on your side ?
---
**Author:**  [@khel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkhel)%5C
**Post date:** [August 24, 2022, 5:31am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsupport-of-very-large-dataset%2F6872%2F13 "2022-08-24T05:31:03Z")
Sorry for the late reply. I missed the thread š.
Last time I looked Datasets codes about this issue, [This](https://github.com/huggingface/datasets/blob/60bdf3005d1dc0b26da8e5949721b20d932eaad6/src/datasets/table.py#L51) takes a lot of time when handling large dataset.