# How can I multithreadedly download a HuggingFace dataset?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-can-i-multithreadedly-download-a-huggingface-dataset%2F56178%3E%5C
**Category:** 🤗Datasets\
**Created:** [September 24, 2023, 6:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-can-i-multithreadedly-download-a-huggingface-dataset%2F56178 "2023-09-24T18:41:59Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Franck-Dernoncourt](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/franck-dernoncourt/32/15950_2.png) [@Franck-Dernoncourt](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FFranck-Dernoncourt)%5C
**Post date:** [September 24, 2023, 6:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-can-i-multithreadedly-download-a-huggingface-dataset%2F56178%2F1 "2023-09-24T18:41:59Z")

</div>

I want to download a HuggingFace dataset, e.g. [`uonlp/CulturaX`](https://huggingface.co/datasets/uonlp/CulturaX):

```auto
from datasets import load_dataset
ds = load_dataset("uonlp/CulturaX", "en")

```

However, it downloads on one thread at 50 MB/s, while my network is 10 Gbps. Since this dataset is 16 TB, I’d prefer to download it faster so that I don’t have to wait for a few days. How can I multithreadedly download a HuggingFace dataset?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [September 25, 2023, 9:42am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-can-i-multithreadedly-download-a-huggingface-dataset%2F56178%2F2 "2023-09-25T09:42:14Z")

</div>

You can use multiprocessing to parallelize the downloads and conversion to Arrow by passing `num_proc=` to `load_dataset`.

---

<div class="post-metadata">

**Author:** ![kopyl](https://avatars.discourse-cdn.com/v4/letter/k/3bc359/32.png) [@kopyl](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkopyl)%5C
**Post date:** [December 19, 2023, 4:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-can-i-multithreadedly-download-a-huggingface-dataset%2F56178%2F3 "2023-12-19T04:29:07Z")

</div>

It does not work. Why?