# Multiprocessing map taking too much memory footprint

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmultiprocessing-map-taking-too-much-memory-footprint%2F27238%3E%5C
**Category:** 🤗Datasets\
**Created:** [December 5, 2022, 2:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmultiprocessing-map-taking-too-much-memory-footprint%2F27238 "2022-12-05T02:50:50Z")\
**Posts on this page:** 1\
**Showing post:** 16

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [April 3, 2024, 12:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmultiprocessing-map-taking-too-much-memory-footprint%2F27238%2F16 "2024-04-03T12:37:24Z")

</div>

> [@StephennFernandes](#):
>
> hey could you elaborate on the args `cache_file_name` what is this arg about and how does it help in memory not blowing up.

A dataset that comes from memory (e.g. using `.from_dict()`) doesn’t have a cache file yet, so if you want your `map()` to write on disk instead of filling up your memory you should pass a `cache_file_name` to `map()`.

Note that at one point we might allocate a cache automatically to such datasets in memory to align with the general behavior.

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmultiprocessing-map-taking-too-much-memory-footprint%2F27238)._