# Eval freezes on local multi GPU Deepspeed run
**URL:**
**Author:**  [@qqhann](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fqqhann)%5C
**Post date:** [April 26, 2021, 9:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Feval-freezes-on-local-multi-gpu-deepspeed-run%2F5762%2F1 "2021-04-26T09:05:58Z")
## Environment info
- `transformers` version: 4.6.0.dev0
- Platform: Linux-4.19.112±x86\_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: \<2,4\>
- Using distributed or parallel set-up in script?:
## Information
I’m working on wav2vec2.0 using the following official script of huggingface.
[https://github.com/huggingface/transformers/blob/master/examples/research\_projects/wav2vec2/run\_common\_voice.py](https://github.com/huggingface/transformers/blob/master/examples/research_projects/wav2vec2/run_common_voice.py)
I am trying to finetune huggingface model with multiple gpus using deepspeed.
```auto
deepspeed --num_gpus=1 run_common_voice.py --deepspeed ds_config.json --do_train --do_eval
```
works, but
```auto
deepspeed --num_gpus=2 run_common_voice.py --deepspeed ds_config.json --do_train --do_eval
```
stops working and freezes at the end of eval.
The progress bar is 100% done but the eval result is not returned and it freezes.
## To reproduce
This is how to reproduce!
> **[Google Colab](https://colab.research.google.com/drive/1VRCGcnhBlrMFYQ5aaNebucZuja-WB2I2?usp=sharing)**
Steps to reproduce the behavior:
1. Install deepspeed
2. Add `with autocast():`after line 481 in run\_common\_voice.py
3. Set param: `--deepspeed ds_config.json --do_train --do_eval`
4. Run run\_common\_voice.py using deepspeed with 1\> gpus
ds\_config has the following parameters.
```auto
{
"fp16": {
"enabled": "true",
"loss_scale": 0,
"loss_scale_window": 1000,
"hysteresis": 2,
"min_loss_scale": 1,
"opt_level": "O3"
},
"steps_per_print": 100,
"wall_clock_breakdown": "false"
}
```
## Expected behavior
The finetuning eval should be executed without freezing.
---
**Author:**  [@qqhann](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fqqhann)%5C
**Post date:** [April 26, 2021, 9:20am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Feval-freezes-on-local-multi-gpu-deepspeed-run%2F5762%2F2 "2021-04-26T09:20:56Z")
This is how to reproduce!
> **[Google Colaboratory](https://colab.research.google.com/drive/1VRCGcnhBlrMFYQ5aaNebucZuja-WB2I2?usp=sharing)**
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [April 26, 2021, 2:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Feval-freezes-on-local-multi-gpu-deepspeed-run%2F5762%2F3 "2021-04-26T14:36:06Z")
cc @stas for deepspeed.
---
**Author:**  [@stas](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fstas)%5C
**Post date:** [April 26, 2021, 4:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Feval-freezes-on-local-multi-gpu-deepspeed-run%2F5762%2F4 "2021-04-26T16:53:49Z")
deepspeed doesn’t work with `autocast`, it has its own way of dealing with mixed precision, if you look in the `trainer.py` it’s carefully bypassed.
If after removing `autocast` the problem persistes, please let’s use an Issue for debugging problems, so it’s easy to track.
**Edit** : I see it was already filed: [[wav2vec] deepspeed eval bug in the case of \>1 gpus · Issue #11446 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/11446)
Thank you.
---
**Author:**  [@qqhann](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fqqhann)%5C
**Post date:** [April 28, 2021, 8:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Feval-freezes-on-local-multi-gpu-deepspeed-run%2F5762%2F5 "2021-04-28T08:53:47Z")
Thanks for replying here and on GitHub! I’ll make further reply on the issue.