# Caching progress of Dataset.from\_generator

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcaching-progress-of-dataset-from-generator%2F113568%3E%5C
**Category:** 🤗Datasets\
**Created:** [October 23, 2024, 12:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcaching-progress-of-dataset-from-generator%2F113568 "2024-10-23T12:18:35Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![alex-hh](https://avatars.discourse-cdn.com/v4/letter/a/c6cbf5/32.png) [@alex-hh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Falex-hh)%5C
**Post date:** [October 23, 2024, 12:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcaching-progress-of-dataset-from-generator%2F113568%2F1 "2024-10-23T12:18:35Z")

</div>

I’m using Dataset.from\_generator to build large datasets.

Assuming the builder writes incrementally to disk as the dataset is constructed, is there a way to automatically resume in case of an error that disrupts construction?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [October 23, 2024, 2:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcaching-progress-of-dataset-from-generator%2F113568%2F2 "2024-10-23T14:23:25Z")

</div>

Hi ! It’s not currently possible 😕

Maybe you can create multiple Dataset objects, this way if one crashes the others can continue

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [October 24, 2024, 2:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcaching-progress-of-dataset-from-generator%2F113568%2F3 "2024-10-24T02:24:03Z")

</div>

Closed on October 24, 2024, 2:24am UTC

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.