# LoadDataSet pyarrow.lib.ArrowCapacityError

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floaddataset-pyarrow-lib-arrowcapacityerror%2F135328%3E%5C
**Category:** 🤗Datasets\
**Created:** [January 12, 2025, 4:14am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floaddataset-pyarrow-lib-arrowcapacityerror%2F135328 "2025-01-12T04:14:59Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [January 12, 2025, 5:18am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floaddataset-pyarrow-lib-arrowcapacityerror%2F135328%2F4 "2025-01-12T05:18:00Z")

</div>

How about trying .shard()?

> [@How to load a large hf dataset efficiently?](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-large-hf-dataset-efficiently%2F69288%2F3)%3A
>
> Thank you for the reply, following your suggestion is there any way in hf dataset, split into multiple shards to be loaded in parallel?

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floaddataset-pyarrow-lib-arrowcapacityerror%2F135328)._