# Is from\_generator() caching? how to stop it?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-from-generator-caching-how-to-stop-it%2F70013%3E%5C
**Category:** 🤗Datasets\
**Created:** [January 22, 2024, 12:57am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-from-generator-caching-how-to-stop-it%2F70013 "2024-01-22T00:57:48Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![badranx](https://avatars.discourse-cdn.com/v4/letter/b/d6d6ee/32.png) [@badranx](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbadranx)%5C
**Post date:** [January 22, 2024, 12:57am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-from-generator-caching-how-to-stop-it%2F70013%2F1 "2024-01-22T00:57:48Z")

</div>

I’m trying `Dataset.from_generator()` loading function, it is awesome! However, on some runs, it simply skips my changes to the generator function. How to stop it from caching?  
I tried:

> disable\_caching()  
> datasets.set\_caching\_enabled(False)

Thank you

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [January 22, 2024, 6:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-from-generator-caching-how-to-stop-it%2F70013%2F2 "2024-01-22T18:10:20Z")

</div>

Hmm can you share more details about your generator ? It’s not supposed to skip you changes.

We haven’t added the argument to `from_generator()` to ask to always use a new cache yet, feel free to open an issue on GitHub to mention this issue if you’re interested.

Note that every time you modify the `gen_kwargs` passed to `from_generator` it will use a new cache, you can leverage this as a workaround.

---

<div class="post-metadata">

**Author:** ![l-uuz](https://avatars.discourse-cdn.com/v4/letter/l/ecae2f/32.png) [@l-uuz](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fl-uuz)%5C
**Post date:** [June 27, 2025, 11:31am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-from-generator-caching-how-to-stop-it%2F70013%2F3 "2025-06-27T11:31:22Z")

</div>

I was also able to observe this. My generator relies on external data written to the disk so skipping it is not intended. I was able to solve the problem by passing a random value to the generator that is ignored:

```python
from datasets import Dataset
import random

def generator(ignored):
  # ... do something
  pass
dataset = Dataset.from_generator(generator, gen_kwargs={"ignored":random.random()})

```