# Should I shard dataset in distributed training?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fshould-i-shard-dataset-in-distributed-training%2F12494%3E%5C
**Category:** 🤗Datasets\
**Created:** [December 3, 2021, 6:08am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fshould-i-shard-dataset-in-distributed-training%2F12494 "2021-12-03T06:08:22Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![pipi](https://avatars.discourse-cdn.com/v4/letter/p/e9bcb4/32.png) [@pipi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpipi)%5C
**Post date:** [December 3, 2021, 6:08am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fshould-i-shard-dataset-in-distributed-training%2F12494%2F1 "2021-12-03T06:08:22Z")

</div>

as the title, should I shard dataset set with rank?  
for example:

```auto
rank_dataset = dataset.shard(num_shards=training_args.world_size, index=training_args.rank)

```

or Trainer will do that automate?

---

<div class="post-metadata">

**Author:** ![PaulLerner](https://avatars.discourse-cdn.com/v4/letter/p/65b543/32.png) [@PaulLerner](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPaulLerner)%5C
**Post date:** [December 3, 2021, 1:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fshould-i-shard-dataset-in-distributed-training%2F12494%2F2 "2021-12-03T13:24:46Z")

</div>

Same question but without using Trainer.  
Like if I’m using DistributedDataParallel to wrap a model but I’m not training it, just processing the dataset, so using load\_from\_disk and Dataset.map

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [December 3, 2021, 5:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fshould-i-shard-dataset-in-distributed-training%2F12494%2F3 "2021-12-03T17:07:17Z")

</div>

The Trainer does the sharding for you. Same if you use Accelerate with your own training loop.