# How to load a large hf dataset efficiently?
**URL:**
**Author:**  [@aaditya](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Faaditya)%5C
**Post date:** [January 16, 2024, 3:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-large-hf-dataset-efficiently%2F69288%2F1 "2024-01-16T03:52:35Z")
I am trying to load a dataset [axiong/pmc\_oa · Datasets at Hugging Face](https://huggingface.co/datasets/axiong/pmc_oa) The dataset size is around 22 gb and I have ram ~10 GB, the dataset object is stuck at extracting file point
I also tried streaming mode but that’s giving another error.
```auto
from datasets import load_dataset
dataset = load_dataset("axiong/pmc_oa", 'pmc_oa', split='train', streaming=True)
print(next(iter(dataset)))
```
Any suggestion on how to deal with large datasets to load in a better manner?
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [January 16, 2024, 4:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-large-hf-dataset-efficiently%2F69288%2F2 "2024-01-16T16:55:07Z")
This dataset is made of one big 20GB+ zip file and a python file to load the data, which is rather unoptimized for data loading.
Ideally a dataset of this size should be split into multiple shards to be loaded in parallel, and a format suited for big image datasets (we usually recommend WebDataset or Parquet).
Anyway I think it just takes some time to extract the big zip file
---
**Author:**  [@aaditya](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Faaditya)%5C
**Post date:** [January 16, 2024, 7:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-large-hf-dataset-efficiently%2F69288%2F3 "2024-01-16T19:22:51Z")
Thank you for the reply, following your suggestion is there any way in hf dataset, split into multiple shards to be loaded in parallel?
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [January 18, 2024, 12:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-large-hf-dataset-efficiently%2F69288%2F4 "2024-01-18T12:13:32Z")
Saving a dataset on HF using [.push\_to\_hub()](https://huggingface.co/docs/datasets/v2.16.1/en/package_reference/main_classes#datasets.Dataset.push_to_hub) does upload multiple shards.
In particular it splits the dataset in shards of 500MB and uploads each shard as a Parquet file on HF.
It’s also possible to manually get a shard of a dataset using the [.shard()](https://huggingface.co/docs/datasets/v2.16.1/en/package_reference/main_classes#datasets.Dataset.shard) method
---
**Author:**  [@samcaetano](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsamcaetano)%5C
**Post date:** [January 20, 2024, 11:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-large-hf-dataset-efficiently%2F69288%2F5 "2024-01-20T23:14:14Z")
@lhoestq what do you think about loading this dataset in a lazy way, for example loading it as a batch of 64 samples at a time? Is it more inefficient than loading those shards of data?
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [January 22, 2024, 6:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-large-hf-dataset-efficiently%2F69288%2F6 "2024-01-22T18:02:22Z")
It’s generally a good idea indeed if you want to save disk space or to not have to wait to download the full dataset.
For example you can stream the dataset using the `datasets` library, by passing `streaming=True` to `load_dataset()`.
However even in streaming mode, you better have multiple shards in order to do parallel streaming. And ideally use file formats that work well with streaming like WebDataset or Parquet.
@aaditya did try streaming mode on this dataset but the custom loading script of this dataset uses the `jsonlines` package that we don’t support for streaming (we only extend the builtin `open()` for streaming)