# Map fails for more than 4 processes
**URL:**
**Author:**  [@jayjha](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjayjha)%5C
**Post date:** [October 14, 2023, 6:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmap-fails-for-more-than-4-processes%2F58567%2F1 "2023-10-14T18:10:16Z")
Hi, I want to apply a company internal transformer based NLP model on the rows in a hf dataset. It works fine when i use map with num\_proc \<=4. For values grater than it errors out with the following stacktrace:
```
Internal error
```
Traceback (most recent call last):
File “/flow/metaflow/metaflow/cli.py”, line 1172, in main
start(auto\_envvar\_prefix=“METAFLOW”, obj=state)
File “/flow/metaflow/metaflow/\_vendor/click/core.py”, line 829, in **call**
return self.main(args, kwargs)
File “/flow/metaflow/metaflow/\_vendor/click/core.py”, line 782, in main
rv = self.invoke(ctx)
File “/flow/metaflow/metaflow/\_vendor/click/core.py”, line 1259, in invoke
return \_process\_result(sub\_ctx.command.invoke(sub\_ctx))
File “/flow/metaflow/metaflow/\_vendor/click/core.py”, line 1066, in invoke
return ctx.invoke(self.callback, ctx.params)
File “/flow/metaflow/metaflow/\_vendor/click/core.py”, line 610, in invoke
return callback(args, kwargs)
File “/flow/metaflow/metaflow/\_vendor/click/decorators.py”, line 21, in new\_func
return f(get\_current\_context(), args, kwargs)
File “/flow/metaflow/metaflow/cli.py”, line 581, in step
task.run\_step(
File “/flow/metaflow/metaflow/task.py”, line 583, in run\_step
self.\_exec\_step\_function(step\_func)
File “/flow/metaflow/metaflow/task.py”, line 57, in \_exec\_step\_function
step\_function()
File “flow.py”, line 196, in hit\_model
result\_dataset = dataset.map(process\_row,num\_proc=16, batch\_size=30)
File “/usr/local/lib/python3.8/site-packages/datasets/arrow\_dataset.py”, line 592, in wrapper
out: Union[“Dataset”, “DatasetDict”] = func(self, args, kwargs)
File “/usr/local/lib/python3.8/site-packages/datasets/arrow\_dataset.py”, line 557, in wrapper
out: Union[“Dataset”, “DatasetDict”] = func(self, args, kwargs)
File “/usr/local/lib/python3.8/site-packages/datasets/arrow\_dataset.py”, line 3189, in map
for rank, done, content in iflatmap\_unordered(
File “/usr/local/lib/python3.8/site-packages/datasets/utils/py\_utils.py”, line 1387, in iflatmap\_unordered
raise RuntimeError(
RuntimeError: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing.
Below is the code which is a step in a metaflow pipeline. we have 32 cpus and 22gigs of memory.
```auto
dataset = load_dataset('/root/dir/',num_proc=32)
MODEL = Model()
def process_row(row):
try:
result = MODEL.run(row)
except:
result = {}
return result
result_dataset = dataset.map(process_row,num_proc=16, batch_size=30)
```
---
**Author:**  [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [October 17, 2023, 10:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmap-fails-for-more-than-4-processes%2F58567%2F2 "2023-10-17T22:50:29Z")
It’s impossible to recover the exception from a failed `multiprocess` process, so set `num_proc=None` and run the `map` on a smaller subset to get an error causing one (or more) of the processes to die.
`concurrent.futures.ProcessPoolExecutor` can recover such errors but is limited in terms of what it can serialize, so we cannot make the switch.
---
**Author:**  [@jayjha](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjayjha)%5C
**Post date:** [October 27, 2023, 4:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmap-fails-for-more-than-4-processes%2F58567%2F3 "2023-10-27T16:49:00Z")
Hi,
The code runs fine for num\_proc=1 or if I spin up really big machine and not use all the processes.
I think my problem is similar to this [issue](https://github.com/huggingface/datasets/issues/4883) discussed where the memory keeps on increasing. Was this ever resolved? is there a new suggested solution other than [this](https://github.com/huggingface/datasets/issues/4883#issuecomment-1764571714)?
---
**Author:**  [@rokayabn](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frokayabn)%5C
**Post date:** [December 12, 2023, 10:21am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmap-fails-for-more-than-4-processes%2F58567%2F4 "2023-12-12T10:21:15Z")
> [@jayjha](#):
>
> RuntimeError: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing.
hi did you find any solution ?
---
**Author:**  [@Dakhoo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDakhoo)%5C
**Post date:** [January 22, 2024, 1:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmap-fails-for-more-than-4-processes%2F58567%2F5 "2024-01-22T13:04:11Z")
> [@rokayabn](#):
>
> any solution ?
I have the same problem
---
**Author:**  [@boringtaskai](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fboringtaskai)%5C
**Post date:** [May 10, 2024, 4:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmap-fails-for-more-than-4-processes%2F58567%2F6 "2024-05-10T16:21:32Z")
I have the same issue
---
**Author:**  [@blackhole33](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fblackhole33)%5C
**Post date:** [January 23, 2025, 6:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmap-fails-for-more-than-4-processes%2F58567%2F8 "2025-01-23T06:48:48Z")
change num\_proc value, that is, reduce !
---
**Author:**  [@ddnick07](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fddnick07)%5C
**Post date:** [April 9, 2025, 5:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmap-fails-for-more-than-4-processes%2F58567%2F9 "2025-04-09T05:48:05Z")
i had facing another error after changed dataset\_num\_proc = 2 to 1 and the error message was,
BackendCompilerFailed: backend=‘inductor’ raised:
PermissionError: [WinError 5] Access is denied: ‘C:\Users\Bespoke\AppData\Local\Temp\torchinductor\_bespoke\triton\0\tmp.8bdc634d-2535-4980-b1ee-7f7a15d6bbca’ → ‘C:\Users\Bespoke\AppData\Local\Temp\torchinductor\_bespoke\triton\0\C42IDn1tsawzf3xVScRF1c7d9fi0Dt73-uD1a5lcLck’
Set TORCH\_LOGS=“+dynamo” and TORCHDYNAMO\_VERBOSE=1 for more information