# Working with large datasets - cache issues

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fworking-with-large-datasets-cache-issues%2F11687%3E%5C
**Category:** 🤗Datasets\
**Created:** [November 14, 2021, 1:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fworking-with-large-datasets-cache-issues%2F11687 "2021-11-14T13:09:36Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![vitvit](https://avatars.discourse-cdn.com/v4/letter/v/b5e925/32.png) [@vitvit](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvitvit)%5C
**Post date:** [November 14, 2021, 1:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fworking-with-large-datasets-cache-issues%2F11687%2F1 "2021-11-14T13:09:36Z")

</div>

The .map() function creates a cache file 100 times larger then the original dataset file.  
Can this behaviour be somehow avoided?  
Note that it happens when using “num\_proc=48”

---

<div class="post-metadata">

**Author:** ![woqucc](https://avatars.discourse-cdn.com/v4/letter/w/e480ec/32.png) [@woqucc](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwoqucc)%5C
**Post date:** [June 1, 2022, 2:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fworking-with-large-datasets-cache-issues%2F11687%2F2 "2022-06-01T02:54:25Z")

</div>

I also encountered this today with datasets=2.2.1 and I’m looking for a solution for that. I processed a 3.7G JSON file by the map function with ‘num\_proc=32’, and more than 300G arrow files were created in the ‘/tmp’ dictionary. Of course, I got a No space on the device error.