# Ideal batch\_size and writer\_batch\_size for datasets

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fideal-batch-size-and-writer-batch-size-for-datasets%2F27517%3E%5C
**Category:** 🤗Datasets\
**Created:** [December 9, 2022, 4:45am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fideal-batch-size-and-writer-batch-size-for-datasets%2F27517 "2022-12-09T04:45:59Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![sriniv](https://avatars.discourse-cdn.com/v4/letter/s/ea5d25/32.png) [@sriniv](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsriniv)%5C
**Post date:** [December 9, 2022, 4:45am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fideal-batch-size-and-writer-batch-size-for-datasets%2F27517%2F1 "2022-12-09T04:45:59Z")

</div>

Hi,

I have csv files with about 1 million rows containing textual data. I am preprocessing this data and experimenting with both datasets.map and pandas with multiprocessing.

For pandas, I am using number of cores as by batch count ( 1 million/num\_cores is batch size) and process them in parallel. Same is being done with Huggingface datasets as well using batch\_size = 1million/num\_cores. However, huggingface datasets map is slower compared to pandas multiprocessing.

@lhoestq - Is there an ideal batch\_size and/or writer\_batch\_size that I can use to make datasets.map faster than pandas multiprocessing?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [December 9, 2022, 10:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fideal-batch-size-and-writer-batch-size-for-datasets%2F27517%2F2 "2022-12-09T10:19:04Z")

</div>

`datasets` processes data on disk by default, which is a bit slower to read and write than pandas which holds the data in RAM. Though you may pass `keep_in_memory=True` when loading the dataset to also have the data in RAM instead of loading them from disk.

Nevertheless, you can tweak `batch_size` and `writer_batch_size` to control how many examples are passed to your processing functions a at time. But this really depends on your resources and the kind of processing you’re doing.

Some functions are faster when batched, but at the same time you don’t always want to give big batches or you may OOM (especially in computer vision or audio processing).