# Dataset map() creates lot of cache files **URL:** **Author:** ![sriniv](https://avatars.discourse-cdn.com/v4/letter/s/ea5d25/32.png) [@sriniv](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsriniv)%5C **Post date:** [July 22, 2022, 12:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-map-creates-lot-of-cache-files%2F20719%2F1 "2022-07-22T12:03:59Z") I am loading a csv file(about 1.5G) from disk using load\_dataset(). It creates files under cache directory. dataset = load\_dataset(ā€˜csvā€™, data\_files=filepath) When we apply map functions on the datasets like below, the cache size keeps growing df= df.map(preprocess\_1, num\_cores=8) df= df.map(preprocess\_2, num\_cores=8) Is there a way to disable caching on each map() function applied. I tried to disable caching at the datasets level using the following, but it still creates cache files. _from datasets import disable\_caching_ _disable\_caching()_ Is there any solution/workaround to disable caching? ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C **Post date:** [July 27, 2022, 9:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-map-creates-lot-of-cache-files%2F20719%2F2 "2022-07-27T09:29:16Z")
It does create files because it writes the resulting dataset on your disk to reload it from there using memory mapping (and save some RAM). To keep your dataset in memory instead, you can pass `keep_in_memory=True` to `map` When caching is disabled, dataset files are written to temporary directories. ---
**Author:** ![sriniv](https://avatars.discourse-cdn.com/v4/letter/s/ea5d25/32.png) [@sriniv](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsriniv)%5C **Post date:** [August 2, 2022, 7:37am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-map-creates-lot-of-cache-files%2F20719%2F3 "2022-08-02T07:37:42Z")
Thanks, @lhoestq ---
**Author:** ![sriniv](https://avatars.discourse-cdn.com/v4/letter/s/ea5d25/32.png) [@sriniv](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsriniv)%5C **Post date:** [August 29, 2022, 11:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-map-creates-lot-of-cache-files%2F20719%2F4 "2022-08-29T11:05:33Z")
@lhoestq If I am applying multiple .map() operationsā€¦ as in below ds = ds.map(preprocess1, batched=True, num\_proc=8) ds = ds.map(preprocess2, batched=True, num\_proc=8) ds = ds.map(preprocess3, batched=True, num\_proc=8) ds = ds.map(preprocess4, batched=True, num\_proc=8) As mentioned above, It creates lot of cache files at each step. Is there a way we can clean cache files after each map step? Also, if we use keep\_in\_memory=True with num\_proc\>1, it slows down. I am using v1.16.1 and I have certain constraints to upgrade. Is there a better way to speed up these preprocessing steps without caching lot of files (constraint on disk space), or with keep\_in\_memory=True and num\_proc \> 1 ? ---
**Author:** ![sriniv](https://avatars.discourse-cdn.com/v4/letter/s/ea5d25/32.png) [@sriniv](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsriniv)%5C **Post date:** [September 14, 2022, 5:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-map-creates-lot-of-cache-files%2F20719%2F5 "2022-09-14T17:17:57Z")
@lhoestq - If we use keep\_in\_memory=True with num\_proc\>1, it slows down. I am using v1.16.1 and I have certain constraints to upgrade. Is there a better way to speed up these preprocessing steps with keep\_in\_memory=True and num\_proc \> 1 ? ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C **Post date:** [September 15, 2022, 12:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-map-creates-lot-of-cache-files%2F20719%2F6 "2022-09-15T12:41:54Z")
If the data is in memory, then it currently needs to be copied to the subprocesses (or to use the disk to do so). On the contrary, data loaded from your disk can be reloaded instantaneously thanks to memory mapping. Thatā€™s why starting a parallel `map` is usually faster with data from your disk. It should be possible to use a Plasma store instead of copying all the data to each subprocesses but this isnā€™t implemented ---
**Author:** ![ShoufaChen](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@ShoufaChen](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FShoufaChen)%5C **Post date:** [March 26, 2024, 6:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-map-creates-lot-of-cache-files%2F20719%2F7 "2024-03-26T06:25:45Z")
Hi, @lhoestq, Thanks for your impressive caching mechanisms. However, I believe it is worth issuing a warning when we use cached files due to a scenario I encountered: During my initial process function, there was a bug. After fixing the bug, I found that the program would skip the map process unless I changed the `num_proc=4` parameter. I find this behavior to be unexpected and undesirable. Thanks for considering my viewpoint.