# Multiprocessing map taking too much memory footprint

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmultiprocessing-map-taking-too-much-memory-footprint%2F27238%3E%5C
**Category:** 🤗Datasets\
**Created:** [December 5, 2022, 2:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmultiprocessing-map-taking-too-much-memory-footprint%2F27238 "2022-12-05T02:50:50Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [December 5, 2022, 3:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmultiprocessing-map-taking-too-much-memory-footprint%2F27238%2F3 "2022-12-05T15:41:58Z")

</div>

Hi! What do you get when you access the `cache_files` attribute of your dataset? `map` with multiprocessing can be an issue for in-memory datasets due to data being copied to the subprocesses ([more info](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-map-creates-lot-of-cache-files%2F20719%2F6)).

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmultiprocessing-map-taking-too-much-memory-footprint%2F27238)._