# How to handle streaming datasets with DDP?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-handle-streaming-datasets-with-ddp%2F70474%3E%5C
**Category:** 🤗Datasets\
**Created:** [January 24, 2024, 8:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-handle-streaming-datasets-with-ddp%2F70474 "2024-01-24T20:55:07Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![goku](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/goku/32/23283_2.png) [@goku](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgoku)%5C
**Post date:** [January 24, 2024, 8:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-handle-streaming-datasets-with-ddp%2F70474%2F1 "2024-01-24T20:55:07Z")

</div>

Let’s say I have a dataset with 5 samples with values [1, 2, 3, 4, 5], with 2 GPUs (for DDP) and batch size of 2. This dataset is I`terableDataset` since I am streaming it.

Now I split the dataset using `split_dataset_by_node` to ensure it doesn’t get repeated. And since it’s already splitted, I don’t have to use DistributedSampler?

But in this case I noticed that the:

First iteraton:  
first GPU will get → [1, 2]  
first GPU will get → [3, 4]

Second iteraton:  
first GPU will get → [5]  
first GPU will get → Nothing

which actually creates an issue since in case of `DistributedSampler`, the samples are repeated internally to ensure non of the GPUs at any iteration is missing any data for gradient sync. So my questions are:

1. Here since splitting is happening before hand, how to make sure each GPU get’s a batch at each iteration to avoid gradient sync issues?
2. Do we need to use `DistributedSampler`? If yes, how?
3. If `dataset.n_shards % world_size != 0`, is it possible to shard the streaming dataset on the fly to avoid the case where data is missing?

---

<div class="post-metadata">

**Author:** ![swheeler](https://avatars.discourse-cdn.com/v4/letter/s/b4bc9f/32.png) [@swheeler](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fswheeler)%5C
**Post date:** [January 28, 2024, 7:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-handle-streaming-datasets-with-ddp%2F70474%2F2 "2024-01-28T19:31:44Z")

</div>

I have the same question