# Dataset map() creates lot of cache files

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-map-creates-lot-of-cache-files%2F20719%3E%5C
**Category:** 🤗Datasets\
**Created:** [July 22, 2022, 12:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-map-creates-lot-of-cache-files%2F20719 "2022-07-22T12:03:59Z")\
**Posts on this page:** 1\
**Showing post:** 6

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [September 15, 2022, 12:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-map-creates-lot-of-cache-files%2F20719%2F6 "2022-09-15T12:41:54Z")

</div>

If the data is in memory, then it currently needs to be copied to the subprocesses (or to use the disk to do so). On the contrary, data loaded from your disk can be reloaded instantaneously thanks to memory mapping. That’s why starting a parallel `map` is usually faster with data from your disk.

It should be possible to use a Plasma store instead of copying all the data to each subprocesses but this isn’t implemented

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-map-creates-lot-of-cache-files%2F20719)._