# Distributed data sampling for streaming

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-data-sampling-for-streaming%2F56386%3E%5C
**Category:** 🤗Datasets\
**Created:** [September 26, 2023, 6:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-data-sampling-for-streaming%2F56386 "2023-09-26T06:55:38Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![BodorS](https://avatars.discourse-cdn.com/v4/letter/b/67e7ee/32.png) [@BodorS](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBodorS)%5C
**Post date:** [September 26, 2023, 6:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-data-sampling-for-streaming%2F56386%2F1 "2023-09-26T06:55:38Z")

</div>

I’m reading data using stream and I need to pass the data to a pipeline which is ran in distributed manner, where each processes is expected to handle different batch of data.

when I tried the following

```auto
dataset = load_dataset("oscar-corpus/OSCAR-2301",
                        token= token
                        language="ar", 
                        streaming=True
                        split="train",
                      )

dataloader= iter(DataLoader(dataset, num_workers=5,batch_size = 1000,collate_fn = lambda x: [i for i in x]))

run_pipes(
        inputs= dataloader, # any inputs of type Iterable 
    )

```

it didn’t work, the dataloader was replicated across processes, and processes ended up with the same batch of data.

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [September 26, 2023, 1:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-data-sampling-for-streaming%2F56386%2F2 "2023-09-26T13:33:51Z")

</div>

Hi! You should be able to avoid this data duplication by using `split_dataset_by_node` as explained in [IterableDataset returns duplicated data using PyTorch DDP · Issue #5360 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/issues/5360).

---

<div class="post-metadata">

**Author:** ![BodorS](https://avatars.discourse-cdn.com/v4/letter/b/67e7ee/32.png) [@BodorS](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBodorS)%5C
**Post date:** [October 4, 2023, 9:03am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistributed-data-sampling-for-streaming%2F56386%2F3 "2023-10-04T09:03:53Z")

</div>

Thank you, that solved the issue