# How do i batch in streaming of data set

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-batch-in-streaming-of-data-set%2F153288%3E%5C
**Category:** Intermediate\
**Created:** [May 3, 2025, 12:35am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-batch-in-streaming-of-data-set%2F153288 "2025-05-03T00:35:01Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [May 3, 2025, 2:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-batch-in-streaming-of-data-set%2F153288%2F2 "2025-05-03T02:19:41Z")

</div>

This might be it. The benefit is probably that it will be processed in parallel according to the number of workers?

> [@How do I iterate through \<class 'datasets.dataset\_dict.IterableDatasetDict'\>?](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-iterate-through-class-datasets-dataset-dict-iterabledatasetdict%2F69140)%3A
>
> I tried using load\_dataset with streaming = Ture for tiiuae/falcon-refinedweb dataset. However, I am unsure how I should iterate through the data. I called the iter() method and next() but I get stopiteration error. TIA!

> [@Streaming batched data](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-batched-data%2F21603)%3A
>
> Hello, I’m trying to batch a streaming dataset. This is what I have done so far: coco\_train = load\_dataset("facebook/pmd", use\_auth\_token=hf\_token, name="coco", cache\_dir="/kaggle/working/", streaming=True, split="train") train\_ds = coco\_train.map(batched=True, batch\_size=32) however, doing next(iter(train\_ds)) only returns one instance (and it is quite slow). Is there something I’m missing here? How do you get a batch of data?

> Now you can iterate on batches using

```auto
for batch in train_ds.iter(batch_size=32):
    ...

```

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-batch-in-streaming-of-data-set%2F153288)._