# How to load a large hf dataset efficiently?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-large-hf-dataset-efficiently%2F69288%3E%5C
**Category:** 🤗Datasets\
**Created:** [January 16, 2024, 3:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-large-hf-dataset-efficiently%2F69288 "2024-01-16T03:52:35Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![aaditya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aaditya/32/20855_2.png) [@aaditya](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Faaditya)%5C
**Post date:** [January 16, 2024, 7:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-large-hf-dataset-efficiently%2F69288%2F3 "2024-01-16T19:22:51Z")

</div>

Thank you for the reply, following your suggestion is there any way in hf dataset, split into multiple shards to be loaded in parallel?

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-large-hf-dataset-efficiently%2F69288)._