# Issue of multiprocessing in map function
**URL:**
**Author:**  [@Prajwal-143](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPrajwal-143)%5C
**Post date:** [March 15, 2024, 6:21am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-of-multiprocessing-in-map-function%2F77510%2F1 "2024-03-15T06:21:03Z")
tokenizer = Wav2Vec2CTCTokenizer(r"D:\Work\Speech to text\Dataset\tamil\_voice\Processed csv\vocab.json", unk\_token=“[UNK]”, pad\_token=“[PAD]”, word\_delimiter\_token=“|”)
feature\_extractor = Wav2Vec2FeatureExtractor(feature\_size=1, sampling\_rate=16000, padding\_value=0.0, do\_normalize=True, return\_attention\_mask=True)
processor = Wav2Vec2Processor(feature\_extractor=feature\_extractor, tokenizer=tokenizer)
def prepare\_dataset(batch):
audio = batch[“path”]
```
# batched output is "un-batched"
batch["input_values"] = processor(audio["array"], sampling_rate=audio["sampling_rate"]).input_values[0]
batch["input_length"] = len(batch["input_values"])
with processor.as_target_processor():
batch["labels"] = processor(batch["sentence"]).input_ids
return batch
```
common\_voice\_train = common\_voice\_train.map(prepare\_dataset, remove\_columns=common\_voice\_train.column\_names,num\_proc= 4)
when i try to call the function i get below mentioned error which occurs only if i enter num\_proc \> 1.
How can i solve this error ?
Error :
* * *
RemoteTraceback Traceback (most recent call last)
RemoteTraceback:
“”"
Traceback (most recent call last):
File “C:\Users\Lenovo\anaconda3\envs\torch\lib\site-packages\multiprocess\pool.py”, line 125, in worker
result = (True, func(\*args, \*\*kwds))
File “C:\Users\Lenovo\anaconda3\envs\torch\lib\site-packages\datasets\utils\py\_utils.py”, line 623, in \_write\_generator\_to\_queue
for i, result in enumerate(func(\*\*kwargs)):
File “C:\Users\Lenovo\anaconda3\envs\torch\lib\site-packages\datasets\arrow\_dataset.py”, line 3458, in \_map\_single
example = apply\_function\_on\_filtered\_inputs(example, i, offset=offset)
File “C:\Users\Lenovo\anaconda3\envs\torch\lib\site-packages\datasets\arrow\_dataset.py”, line 3361, in apply\_function\_on\_filtered\_inputs
processed\_inputs = function(\*fn\_args, \*additional\_args, \*\*fn\_kwargs)
File “C:\Users\Lenovo\AppData\Local\Temp\ipykernel\_6536\430615538.py”, line 5, in prepare\_dataset
NameError: name ‘processor’ is not defined
“”"
The above exception was the direct cause of the following exception:
NameError Traceback (most recent call last)
Cell In[23], line 1
----\> 1 common\_voice\_train = common\_voice\_train.map(prepare\_dataset, remove\_columns=common\_voice\_train.column\_names,num\_proc= 4)
2 common\_voice\_test = common\_voice\_test.map(prepare\_dataset, remove\_columns=common\_voice\_test.column\_names)
File ~\anaconda3\envs\torch\lib\site-packages\datasets\arrow\_dataset.py:593, in transmit\_tasks..wrapper(\*args, \*\*kwargs)
591 self: “Dataset” = kwargs.pop(“self”)
592 # apply actual function
→ 593 out: Union[“Dataset”, “DatasetDict”] = func(self, \*args, \*\*kwargs)
594 datasets: List[“Dataset”] = list(out.values()) if isinstance(out, dict) else [out]
595 for dataset in datasets:
596 # Remove task templates if a column mapping of the template is no longer valid
File ~\anaconda3\envs\torch\lib\site-packages\datasets\arrow\_dataset.py:558, in transmit\_format..wrapper(\*args, \*\*kwargs)
551 self\_format = {
552 “type”: self.\_format\_type,
553 “format\_kwargs”: self.\_format\_kwargs,
554 “columns”: self.\_format\_columns,
555 “output\_all\_columns”: self.\_output\_all\_columns,
556 }
557 # apply actual function
→ 558 out: Union[“Dataset”, “DatasetDict”] = func(self, \*args, \*\*kwargs)
559 datasets: List[“Dataset”] = list(out.values()) if isinstance(out, dict) else [out]
560 # re-apply format to the output
File ~\anaconda3\envs\torch\lib\site-packages\datasets\arrow\_dataset.py:3197, in Dataset.map(self, function, with\_indices, with\_rank, input\_columns, batched, batch\_size, drop\_last\_batch, remove\_columns, keep\_in\_memory, load\_from\_cache\_file, cache\_file\_name, writer\_batch\_size, features, disable\_nullable, fn\_kwargs, num\_proc, suffix\_template, new\_fingerprint, desc)
3191 logger.info(f"Spawning {num\_proc} processes")
3192 with hf\_tqdm(
3193 unit=" examples",
3194 total=pbar\_total,
3195 desc=(desc or “Map”) + f" (num\_proc={num\_proc})",
3196 ) as pbar:
→ 3197 for rank, done, content in iflatmap\_unordered(
3198 pool, Dataset.\_map\_single, kwargs\_iterable=kwargs\_per\_job
3199 ):
3200 if done:
3201 shards\_done += 1
File ~\anaconda3\envs\torch\lib\site-packages\datasets\utils\py\_utils.py:663, in iflatmap\_unordered(pool, func, kwargs\_iterable)
660 finally:
661 if not pool\_changed:
662 # we get the result in case there’s an error to raise
→ 663 [async\_result.get(timeout=0.05) for async\_result in async\_results]
File ~\anaconda3\envs\torch\lib\site-packages\datasets\utils\py\_utils.py:663, in (.0)
660 finally:
661 if not pool\_changed:
662 # we get the result in case there’s an error to raise
→ 663 [async\_result.get(timeout=0.05) for async\_result in async\_results]
File ~\anaconda3\envs\torch\lib\site-packages\multiprocess\pool.py:774, in ApplyResult.get(self, timeout)
772 return self.\_value
773 else:
→ 774 raise self.\_value
NameError: name ‘processor’ is not defined
---
**Author:**  [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [March 15, 2024, 4:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-of-multiprocessing-in-map-function%2F77510%2F2 "2024-03-15T16:56:05Z")
This issue looks similar to [I'm trying to fine tune the openai/whisper model from huggingface using jupyter notebook and i keep getting this error · Issue #6276 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/issues/6276). Can you try my suggested fix?
---
**Author:**  [@Prajwal-143](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPrajwal-143)%5C
**Post date:** [March 18, 2024, 4:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-of-multiprocessing-in-map-function%2F77510%2F3 "2024-03-18T04:40:22Z")
Yeah, I tried but it didn’t work.