# Best practices for a large dataset

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbest-practices-for-a-large-dataset%2F137632%3E%5C
**Category:** 🤗Datasets\
**Created:** [January 27, 2025, 11:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbest-practices-for-a-large-dataset%2F137632 "2025-01-27T23:10:20Z")\
**Posts on this page:** 1\
**Showing post:** 8

<div class="post-metadata">

**Author:** ![nitishpandey04](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nitishpandey04/32/33204_2.png) [@nitishpandey04](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnitishpandey04)%5C
**Post date:** [May 6, 2025, 2:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbest-practices-for-a-large-dataset%2F137632%2F8 "2025-05-06T14:21:42Z")

</div>

Tell me, so you have implemented a pytorch `IterableDataset` that is calling 2 huggingface `IterableDataset`s for +ve and -ve class items right?

I think using individual iterable datasets for +ve and -ve classes can be a good idea but the wrapper you are using on top of those datasets doesn’t need to be iterable dataset itself. The parent can be a normal dataset. Have you tried this ?

Or maybe you can try to implement a single iterable dataset in pytorch which loads both +ve and -ve classes

I opened this issue - [Big text dataset loading for training](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbig-text-dataset-loading-for-training%2F153576). Any insights you can share ?

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbest-practices-for-a-large-dataset%2F137632)._