# Creating dataset slow

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcreating-dataset-slow%2F132008%3E%5C
**Category:** 🤗Datasets\
**Created:** [December 18, 2024, 5:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcreating-dataset-slow%2F132008 "2024-12-18T05:34:29Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [December 18, 2024, 5:49am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcreating-dataset-slow%2F132008%2F2 "2024-12-18T05:49:36Z")

</div>

Perhaps the data set has become too large, exceeding the capacity of the RAM, and you are using an SSD or HDD as a substitute for RAM?  
You might want to look for some know-how on creating large data sets.

> [@How to create a new large Dataset on disk?](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-a-new-large-dataset-on-disk%2F19616)%3A
>
> I have my own library which processes corpora of documents much larger than what fits into memory. I would like to create a new HF Dataset by incrementally adding/streaming new examples to it, but because of the number of examples the dataset would not fit into memory. I am not sure if add\_item should get used for this as it does not seem to modify the dataset in place but rather returns a new dataset which looks like the dataset would actually be immutable and add\_item would create a new data…

> [@Strategy for generating a large dataset](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstrategy-for-generating-a-large-dataset%2F29063)%3A
>
> Hi, I have to generate a dataset from 1,000+ large files by: making a random choice with replacement of a file per example (fast, this step takes a total of ~1 min for all examples). We need to keep a list of labels per file that describe some categories the file belongs to. sampling each of the chosen files at a random location (slow, ~a few days) and extracting a numerical vector per example Some constraints: the data is proprietary and the dataset cannot be uploaded online, it has to rem…

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcreating-dataset-slow%2F132008)._