# Limitations of iterable datasets

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flimitations-of-iterable-datasets%2F16794%3E%5C
**Category:** 🤗Datasets\
**Created:** [April 13, 2022, 1:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flimitations-of-iterable-datasets%2F16794 "2022-04-13T13:54:45Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![adrienchaton](https://avatars.discourse-cdn.com/v4/letter/a/41988e/32.png) [@adrienchaton](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fadrienchaton)%5C
**Post date:** [April 25, 2022, 12:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flimitations-of-iterable-datasets%2F16794%2F4 "2022-04-25T12:32:32Z")

</div>

Hi @mariosasko

I am actually having an idea why the loss would behave differently in streaming and non-streaming mode, it would be great if you could confirm please.  
When I am training with streaming (i.e. iterable dataset), the logger only sees one epoch which is the chosen number of training steps.  
Then I am afraid there is no reshuffling of the dataset during training … am I right ?

Question here is what is the best way to fix this please ?  
Is there a place where I should configure the length of the dataset, which is known in advance in my case ?  
Or should I make a callback every length dataset / batch size to manually shuffle the dataset ?

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flimitations-of-iterable-datasets%2F16794)._