# How to save datasets as distributed with save\_to\_disk?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-save-datasets-as-distributed-with-save-to-disk%2F25674%3E%5C
**Category:** 🤗Datasets\
**Created:** [November 7, 2022, 10:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-save-datasets-as-distributed-with-save-to-disk%2F25674 "2022-11-07T10:46:07Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![fujindemi](https://avatars.discourse-cdn.com/v4/letter/f/7cd45c/32.png) [@fujindemi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffujindemi)%5C
**Post date:** [November 7, 2022, 10:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-save-datasets-as-distributed-with-save-to-disk%2F25674%2F1 "2022-11-07T10:46:07Z")

</div>

I have a lot of data to save. Is it possible to save data into multiple files and load multiple files together?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [November 15, 2022, 5:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-save-datasets-as-distributed-with-save-to-disk%2F25674%2F2 "2022-11-15T17:54:21Z")

</div>

Hi ! Right now you have to shard the dataset yourself to save multiple files, but I’m working on supporting saving into multiple files, it will be available soon 😉

In the meantime you can do:

```python
ds = load_dataset(...)
num_shards = 32
for shard_idx in range(num_shards):
    shard = ds.shard(num_shards=num_shards, index=shard_idx, contiguous=True)
    shard.save_to_disk(f"path/to/shard_{shard_idx}")

# reload later
from datasets import load_from_disk, concatenate_datasets
ds = concatenate_datasets([
    load_from_disk(f"path/to/shard_{shard_idx}")
    for shard_idx in range(num_shards)
])
```