# Dataset.map() with batching and multiprocessing

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-map-with-batching-and-multiprocessing%2F75439%3E%5C
**Category:** 🤗Datasets\
**Created:** [March 1, 2024, 6:57am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-map-with-batching-and-multiprocessing%2F75439 "2024-03-01T06:57:42Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![varadhbhatnagar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/varadhbhatnagar/32/31383_2.png) [@varadhbhatnagar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvaradhbhatnagar)%5C
**Post date:** [March 1, 2024, 6:57am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-map-with-batching-and-multiprocessing%2F75439%2F1 "2024-03-01T06:57:42Z")

</div>

I am seeing different results when I do

`dataset.map(..., batched=True, num_proc=4)`  
vs  
`dataset.map(..., batched=True, num_proc=16)`

Here is the output:

```auto
Map (num_proc=4): 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [00:00<00:00, 1019.49 examples/s]
Map (num_proc=16): 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [00:00<00:00, 1684.70 examples/s]
Dataset({
    features: ['input_ids', 'attention_mask'],
    num_rows: 148
}) Dataset({
    features: ['input_ids', 'attention_mask'],
    num_rows: 143
})

```

Is it expected to have different `num_rows` in the output?

---

<div class="post-metadata">

**Author:** ![varadhbhatnagar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/varadhbhatnagar/32/31383_2.png) [@varadhbhatnagar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvaradhbhatnagar)%5C
**Post date:** [March 5, 2024, 6:31am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-map-with-batching-and-multiprocessing%2F75439%2F2 "2024-03-05T06:31:41Z")

</div>

@muellerzr Any thoughts on this?