# Wav2vec fine-tuning with multiGPU

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-with-multigpu%2F4894%3E%5C
**Category:** Models\
**Created:** [March 22, 2021, 1:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-with-multigpu%2F4894 "2021-03-22T13:50:05Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![gorodecki](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@gorodecki](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgorodecki)%5C
**Post date:** [March 22, 2021, 1:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-with-multigpu%2F4894%2F1 "2021-03-22T13:50:05Z")

</div>

Hi, @patrickvonplaten @valhalla  
I’m fine-tuning wav2vec model with [Fine-Tune XLSR-Wav2Vec2 for low-resource ASR with 🤗 Transformers](https://huggingface.co/blog/fine-tune-xlsr-wav2vec2) at local machine with 4xT4 GPU (16Gb)  
I have some problems with training.

1. Very slowly process ![use_one_gpu](https://us1.discourse-cdn.com/hellohellohello/original/2X/c/c11e8bb5b35217f29ca9247ee48c1c4dba30629b.png)

![use_3_gpu](https://us1.discourse-cdn.com/hellohellohello/original/2X/e/e72ac8b63513af8042b25b6e67975868e5dfadfb.png)

Why has the learning process slowed down so much?

---

<div class="post-metadata">

**Author:** ![gorodecki](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@gorodecki](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgorodecki)%5C
**Post date:** [March 22, 2021, 2:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-with-multigpu%2F4894%2F2 "2021-03-22T14:26:47Z")

</div>

I noticed a strange memory allocation on the GPU

 ![tesla_t4_4](https://us1.discourse-cdn.com/hellohellohello/original/2X/1/1908242168ed525cd03a808f6c21895d6b3679a6.png)

```
training_args = TrainingArguments(
    output_dir="./rus_model",
    group_by_length=True,
    per_device_train_batch_size=16,
    gradient_accumulation_steps=2,
    evaluation_strategy="steps",
    num_train_epochs=30,
    fp16=True,
    save_steps=400,
    eval_steps=200,
    logging_steps=100,
    learning_rate=3e-4,
    warmup_steps=500,
    save_total_limit=2,
    dataloader_num_workers=16,
    report_to='tensorboard'
)

```

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [March 23, 2021, 3:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-with-multigpu%2F4894%2F3 "2021-03-23T15:58:31Z")

</div>

Related issue: [Excessive GPU-GPU communication with GPT2 making multi-GPU training slow? · Issue #9371 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/9371)

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [March 23, 2021, 4:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-with-multigpu%2F4894%2F4 "2021-03-23T16:03:53Z")

</div>

The GPU allocation is not unreasonable since one of the 4 GPUs has to store all optimizer state since the gradient updates are always done on only one of the four GPUs.

I have never seen such a heavy slow down when using multiple GPUs. Note also though, that if you leave `per_device_train_batch_size=16` when doing multi GPU training you increase your effective batch size by a number of 4 in your setup. But I’m not really sure what’s going on here…what happens if you reduce your batch\_size?

Finally, it is recommend by PyTorch to do distributed training instead of “multi-GPU” training. Here is an example on how to do distributed training: [transformers/examples at master · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/tree/master/examples#distributed-training-and-mixed-precision)

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [March 23, 2021, 4:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-with-multigpu%2F4894%2F5 "2021-03-23T16:05:39Z")

</div>

Also gently pinging @sgugger in case my answers are inaccurate or you see a bug, I’m not seeing 🙂

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [March 23, 2021, 4:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-with-multigpu%2F4894%2F6 "2021-03-23T16:58:29Z")

</div>

I haven’t studied `DataParallel` enough to know why this is happening or if your explanations are inacurate. I only use the distributed one since it is what PyTorch recommends.

---

<div class="post-metadata">

**Author:** ![Srulikbdd](https://avatars.discourse-cdn.com/v4/letter/s/c37758/32.png) [@Srulikbdd](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSrulikbdd)%5C
**Post date:** [March 24, 2021, 3:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-with-multigpu%2F4894%2F7 "2021-03-24T15:24:38Z")

</div>

it helped me to choose less GPUs to use via  
export CUDA\_VISIBLE\_DEVICES=1,2,3

---

<div class="post-metadata">

**Author:** ![gorodecki](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@gorodecki](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgorodecki)%5C
**Post date:** [March 24, 2021, 9:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-with-multigpu%2F4894%2F8 "2021-03-24T21:21:08Z")

</div>

Thank you!  
I’m start [run\_common\_voice.py](https://github.com/huggingface/transformers/blob/master/examples/research_projects/wav2vec2/run_common_voice.py) with `torch.distributed.launch` and all 4 GPU the loading normal.  
I had to set the size `per_device_train_batch_size="3"`, otherwise the system would give an error CUDA: out\_of\_memory. 😱 Why? What is the acceleration from this use of multiGPU, in fact, the size of the total batch is 12? 🤗 Where am I going wrong?

---

<div class="post-metadata">

**Author:** ![gorodecki](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@gorodecki](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgorodecki)%5C
**Post date:** [March 26, 2021, 11:37am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-with-multigpu%2F4894%2F9 "2021-03-26T11:37:08Z")

</div>

I start training my tuned in jupyter model with multiGPU with this script `run_common_voice.py` i see this error:

> RuntimeError: Expected to mark a variable ready only once.  
> This error is caused by one of the following reasons: 1) Use of a module parameter outside the `forward` function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes  
> 2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple `checkpoint` functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases yet.

Traceback (most recent call last):  
File “run\_common\_voice\_rus.py”, line 434, in   
main()  
File “run\_common\_voice\_rus.py”, line 402, in main  
train\_result = trainer.train(resume\_from\_checkpoint=checkpoint)  
File “/lib/python3.8/site-packages/transformers/trainer.py”, line 1056, in train  
tr\_loss += self.training\_step(model, inputs)  
File “run\_common\_voice\_rus.py”, line 247, in training\_step  
self.scaler.scale(loss).backward()  
File “/lib/python3.8/site-packages/torch/tensor.py”, line 245, in backward  
torch.autograd.backward(self, gradient, retain\_graph, create\_graph, inputs=inputs)  
File “/lib/python3.8/site-packages/torch/autograd/ **init**.py”, line 145, in backward  
Variable.\_execution\_engine.run\_backward(  
File “/lib/python3.8/site-packages/torch/autograd/function.py”, line 89, in apply  
return self.\_forward\_cls.backward(self, \*args) # type: ignore  
File “lib/python3.8/site-packages/torch/utils/checkpoint.py”, line 112, in backward  
torch.autograd.backward(outputs\_with\_grad, args\_with\_grad)  
File “/lib/python3.8/site-packages/torch/autograd/ **init**.py”, line 145, in backward  
Variable.\_execution\_engine.run\_backward(  
@sgugger  
What is the reason for this error? Why does it appear on the pre-train model?

---

<div class="post-metadata">

**Author:** ![gorodecki](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@gorodecki](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgorodecki)%5C
**Post date:** [March 31, 2021, 4:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-with-multigpu%2F4894%2F10 "2021-03-31T16:33:36Z")

</div>

@patrickvonplaten @sgugger Can you help in solving this issue? 🙏 👀

---

<div class="post-metadata">

**Author:** ![Maimonator](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/maimonator/32/1452_2.png) [@Maimonator](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMaimonator)%5C
**Post date:** [April 11, 2021, 10:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-with-multigpu%2F4894%2F11 "2021-04-11T10:15:12Z")

</div>

Don’t know if this is still relevant but I’ve had a similar issue using Multi-GPU so after a lot of googling I found these:

> **[Fit More and Train Faster With ZeRO via DeepSpeed and FairScale](https://huggingface.co/blog/zero-deepspeed-fairscale)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

> **[Trainer](https://huggingface.co/docs/transformers/main/main_classes/trainer)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Using deepspeed helped me accelerate my training procedure, and also make sure you use `shared_ddp` to make sure memory is evenly distributed among GPUs.

Hopefully that helps

---

<div class="post-metadata">

**Author:** ![tommy19970714](https://avatars.discourse-cdn.com/v4/letter/t/e8c25b/32.png) [@tommy19970714](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftommy19970714)%5C
**Post date:** [April 14, 2021, 1:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-with-multigpu%2F4894%2F12 "2021-04-14T13:41:07Z")

</div>

@Maimonator Can you tell me how to set the parameter of `shared_ddp`?

I have tried to use deepspeed. But I occurred error.  
That error is the same as the following issue (`RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one...`).

Can you tell me your json of deepspeed config?

> <https://github.com/huggingface/transformers/issues/10878>
>
> \## Environment info
> \<!-- You can run the command \`transformers-cli env\` and cop…y-and-paste its output below.
> Don't forget to fill out the missing fields in that output! --\>
> 
> \- \`transformers\` version: 4.5.0.dev0 (I tried running it on 4.4.0 as well, gave the same error)
> \- Platform: Ubuntu (running on a virtual machine)
> \- Python version: 3.8
> \- PyTorch version (GPU?): 1.6.0
> \- Using GPU in script?: yes, running \[this script\](https://github.com/huggingface/transformers/blob/master/examples/research\_projects/wav2vec2/run\_common\_voice.py)
> \- Using distributed or parallel set-up in script?: Distributed 
> 
> \### Who can help
> @patrickvonplaten (as per the message on slack group)
> \<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
> If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of \*\*who to tag\*\*.
> Please tag fewer than 3 people.
> 
> Models:
> 
> \- albert, bert, xlm: @LysandreJik
> \- blenderbot, bart, marian, pegasus, encoderdecoder, t5: @patrickvonplaten, @patil-suraj
> \- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
> \- fsmt: @stas00
> \- funnel: @sgugger
> \- gpt2: @patrickvonplaten, @LysandreJik
> \- rag: @patrickvonplaten, @lhoestq
> \- tensorflow: @LysandreJik
> 
> Library:
> 
> \- benchmarks: @patrickvonplaten
> \- deepspeed: @stas00
> \- ray/raytune: @richardliaw, @amogkam
> \- text generation: @patrickvonplaten
> \- tokenizers: @LysandreJik
> \- trainer: @sgugger
> \- pipelines: @LysandreJik
> 
> Documentation: @sgugger
> 
> Model hub:
> 
> \- for issues with a model report at /static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2F and tag the model's creator.
> 
> HF projects:
> 
> \- nlp datasets: \[different repo\](https://github.com/huggingface/nlp)
> \- rust tokenizers: \[different repo\](https://github.com/huggingface/tokenizers)
> 
> Examples:
> 
> \- maintained examples (not research project or legacy): @sgugger, @patil-suraj
> \- research\_projects/bert-loses-patience: @JetRunner
> \- research\_projects/distillation: @VictorSanh
> 
> --\>
> 
> \## Information
> 
> Model I am using (Bert, XLNet ...):
> 
> The problem arises when using:
> 
> \- \[\] the official example scripts: (give details below)   
> \- \[\] my own modified scripts: (give details below)
> 
> Tried running both official command and modified script (running command changed based on the language) 
> 
> The tasks I am working on is
> \- \[\] common voice dataset (ta)
> 
> \## To reproduce
> 
> Steps to reproduce the behavior:
> 
> 1. run common voice script \[from here\](https://github.com/huggingface/transformers/blob/master/examples/research\_projects/wav2vec2/run\_common\_voice.py)
> 2. For multi-gpu setup I used this command \`python -m torch.distributed.launch \\
> --nproc\_per\_node 4 run\_common\_voice.py \\
> --model\_name\_or\_path="facebook/wav2vec2-large-xlsr-53" \\
> --dataset\_config\_name="tr" \\ # use this argument to specify the language code
> --output\_dir=./wav2vec2-large-xlsr-turkish-demo \\
> --overwrite\_output\_dir \\
> --num\_train\_epochs="5" \\
> --per\_device\_train\_batch\_size="16" \\
> --learning\_rate="3e-4" \\
> --warmup\_steps="500" \\
> --evaluation\_strategy="steps" \\
> --save\_steps="400" \\
> --eval\_steps="400" \\
> --logging\_steps="400" \\
> --save\_total\_limit="3" \\
> --freeze\_feature\_extractor \\
> --feat\_proj\_dropout="0.0" \\
> --layerdrop="0.1" \\
> --gradient\_checkpointing \\
> --fp16 \\
> --group\_by\_length \\
> --do\_train --do\_eval \`
> 
> \## Error:
> \`RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument 'find\_unused\_parameters=True' to 'torch.nn.parallel.DistributedDataParallel'; (2) making sure all 'forward' function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's 'forward' function. Please include the loss function and the structure of the return value of 'forward' of your module when reporting this issue (e.g. list, dict, iterable).\`
> 
> \<!-- If you have code snippets, error messages, stack traces please provide them here as well.
> Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
> Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.--\>
> 
> \## Expected behavior
> Model would train without any error

---

<div class="post-metadata">

**Author:** ![Maimonator](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/maimonator/32/1452_2.png) [@Maimonator](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMaimonator)%5C
**Post date:** [April 14, 2021, 4:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-with-multigpu%2F4894%2F13 "2021-04-14T16:03:20Z")

</div>

@tommy19970714  
To be honest I’ve found success using the default parameters on my use-case so I never got the chance to dive deeper into the deepspeed configuration, but I found this that might be helpful to you.

> <https://github.com/huggingface/transformers/issues/10011>
>
> I'm currently following this \[notebook\](https://colab.research.google.com/drive/…12LjJazBl7Gam0XBPy\_y0CTOJZeZ34c2v?usp=sharing#scrollTo=tLM3niQqhEzP) but instead I'm using \`patrickvonplaten/led-large-16384-pubmed\`
> \`\`\`python
> 
> tokenizer = AutoTokenizer.from\_pretrained("patrickvonplaten/led-large-16384-pubmed",)
> 
> led = AutoModelForSeq2SeqLM.from\_pretrained(
> "patrickvonplaten/led-large-16384-pubmed",
> gradient\_checkpointing=True,
> use\_cache=False,
> )
> \`\`\`
> 
> instead of \`allenai/led-large-16384\` as the base model and tokenizer. I'm also using my own train/test data. With the exception of that, I kept everything else the same/consistent to that notebook as far as fine tuning. However, I'm running into OOM errors
> 
> \`\`\`
> RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 15.78 GiB total capacity; 13.96 GiB already allocated; 20.00 MiB free; 14.56 GiB reserved in total by PyTorch)
> 
> 0%| | 0/3 \[00:10\<?, ?it/s\]
> \`\`\`
> on a couple of\`Tesla V100-SXM2-16GB\` and I'm not sure why that might be. The \`batch\_size=2\` seems pretty small and I also set \`gradient\_checkpoint=True\`. @patrickvonplaten and/or the surrounding community, I'd greatly appreciate any help with this

---

<div class="post-metadata">

**Author:** ![tommy19970714](https://avatars.discourse-cdn.com/v4/letter/t/e8c25b/32.png) [@tommy19970714](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftommy19970714)%5C
**Post date:** [April 14, 2021, 4:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-with-multigpu%2F4894%2F14 "2021-04-14T16:21:41Z")

</div>

@Maimonator Thank you for your reply! The issue you shared was very helpful.

Did you run your experiment without the deepspeed config?

Then I experimented, and the following config worked(In this case, I used huggingface optimizer).

```auto
{
  "fp16": {
    "enabled": true,
    "min_loss_scale": 1,
    "opt_level": "O3"
  }
}

```

However, the next config gave me the following error. (In this case, I used deepspeed optimizer)  
Did you use the huggingface optimaizer or the deepspeed optimizer?

```auto
{
  "fp16": {
    "enabled": true,
    "min_loss_scale": 1,
    "opt_level": "O3"
  },
  "zero_optimization": {
    "stage": 3,
    "cpu_offload": true,
    "cpu_offload_params": true,
    "contiguous_gradients": true,
    "overlap_comm": true
  },
  "optimizer": {
    "type": "AdamW",
    "params": {
      "lr": 0.001,
      "betas": [
        0.9,
        0.999
      ],
      "eps": 1e-6
    }
  },
  "scheduler": {
    "type": "WarmupLR",
    "params": {
      "warmup_min_lr": 0,
      "warmup_max_lr": 0.001,
      "warmup_num_steps": 50
    }
  }
} 

```

```auto
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 2)

```

---

<div class="post-metadata">

**Author:** ![liaspas](https://avatars.discourse-cdn.com/v4/letter/l/a87d85/32.png) [@liaspas](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fliaspas)%5C
**Post date:** [April 17, 2021, 1:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-with-multigpu%2F4894%2F16 "2021-04-17T13:19:19Z")

</div>

Also facing issues with multiGPU training.  
Trying to finetune with DistributedDataParallel gives me the following error.

> RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making sure all `forward` function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn’t able to locate the output tensors in the return value of your module’s `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).

If I set find\_unused\_parameters=True then I get the same error as @gorodecki

> RuntimeError: Expected to mark a variable ready only once.  
> This error is caused by one of the following reasons: 1) Use of a module parameter outside the `forward` function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes  
> 2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple `checkpoint` functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases yet.

All is fine with one GPU however.

---

<div class="post-metadata">

**Author:** ![stas](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stas/32/19289_2.png) [@stas](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fstas)%5C
**Post date:** [May 10, 2021, 4:40pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-with-multigpu%2F4894%2F17 "2021-05-10T16:40:34Z")

</div>

Here is WIP PR that makes it work with deepspeed:

> <https://github.com/huggingface/transformers/pull/11638>
>
> Addressing the need in https://github.com/huggingface/transformers/issues/11446,… this PR is working on making wav2vec2 work under deepspeed.
> 
> This PR:
> \* changes Trainer to automatically convert inputs to the correct dtype if it's not int64 - we didn't need this for nlp models because embeddings took care of this - this is not the case with wav2vec2 type of models where inputs are float32 by default. (for deepspeed only at the moment - potentially need to do the same for \`fp16\_full\_eval\`)
> \* multiple fixes to the \`wav2vec2\` model, because it does very non-standard things, like model \`weight\_norm\` which is implemented in a very odd way and deepspeed's automatic ease-of-use fails to do that and requires multiple manual adjustments for it to do the right thing. \`weight\_norm\` creates a param, then drops it replacing it with 2 other params and re-creates them on every forward. (in pre-hook).
> \* moves \`require\_deepspeed\` to \`testing\_utils.py\` as we have multiple test files using it
> \* adds \`dtype\` accessor to DS conf object
> \* adds 8 new tests, checking each setup
> 
> Testing with \`run\_asr.py\`:
> 
> \### ZeRO-2 
> 
> Everything works:
> 
> \* \[x\] fp16 distributed zero2
> \* \[x\] fp16 non distributed zero2
> \* \[x\] fp32 distributed zero2
> \* \[x\] fp32 non distributed zero2
> 
> important - must use for distributed use:
> \`\`\`
> "zero\_optimization": {
> "find\_unused\_parameters": true,
> \`\`\`
> So you can use the \`--deepspeed examples/research\_projects/wav2vec2/ds\_config\_wav2vec2\_zero2.json\` which already has the adjustment.
> 
> \### ZeRO-3 
> 
> You can use the \`--deepspeed examples/research\_projects/wav2vec2/ds\_config\_wav2vec2\_zero3.json\`
> 
> This works:
> 
> \* \[x\] fp32 non distributed zero3
> \* \[x\] fp32 distributed zero3 
> \* \[x\] fp16 non distributed zero3
> \* \[x\] fp16 distributed zero3
> 
> 
> \### Possible PR spin-offs
> 
> it looks like plain pytorch dist doesn't work either https://github.com/huggingface/transformers/issues/11452
> so this PR can be adapted to detect \`dist\` and do the same as what deepspeed branch does. probably a separate PR is the best.
> (LayerSkip that is)
> 
> \---------------------
> 
> To run tests:
> 
> Install this deepspeed master https://github.com/microsoft/DeepSpeed:
> \`\`\`
> pip install deepspeed
> \`\`\`
> 
> and then:
> 
> \`\`\`
> HF\_DATASETS\_IN\_MEMORY\_MAX\_SIZE=0 RUN\_SLOW=1 pyt examples/research\_projects/wav2vec2/test\_wav2vec2\_deepspeed.py
> 
> \`\`\`
> 
> \---------------------
> Example of usage: assuming you in a top dir of the git clone of this branch
> 
> \### \`run\_asr.py\` and tiny model and tiny dataset
> 
> This is the foundation for the new tests:
> 
> \`\`\`
> HF\_DATASETS\_IN\_MEMORY\_MAX\_SIZE=0 PYTHONPATH=src deepspeed --num\_gpus 2 \\
> examples/research\_projects/wav2vec2/run\_asr.py \\
> \--output\_dir=output\_dir --num\_train\_epochs=2 --per\_device\_train\_batch\_size=2 \\
> \--per\_device\_eval\_batch\_size=2 --evaluation\_strategy=steps --save\_steps=500 --eval\_steps=100 \\
> \--logging\_steps=5 --learning\_rate=5e-4 --warmup\_steps=3000 \\
> \--model\_name\_or\_path=patrickvonplaten/wav2vec2\_tiny\_random\_robust \\
> \--dataset\_name=patrickvonplaten/librispeech\_asr\_dummy --dataset\_config\_name=clean \\
> \--train\_split\_name=validation --validation\_split\_name=validation --orthography=timit \\
> \--preprocessing\_num\_workers=1 --group\_by\_length --freeze\_feature\_extractor --verbose\_logging \\
> \--deepspeed examples/research\_projects/wav2vec2/ds\_config\_wav2vec2\_zero2.json
> \`\`\`
> 
> \### run\_common\_voice.py
> 
> very hard to test with as it takes some 5-10mins to just get ready to run.
> 
> \*\*edit\*\*: switch to \`datasets\` master branch and add \`HF\_DATASETS\_IN\_MEMORY\_MAX\_SIZE=0\` to the command line - it will be cached now.
> 
> \`run\_common\_voice.py\` now runs under \`--fp16\` but gives \`loss=nan\`, probably the same issue as bf16-pretrained models? I tested - it has the same issue under AMP and no deepspeed. So it's a different problem to solve.
> 
> fp32 works just fine loss-wise, you can try:
> 
> \`\`\`
> HF\_DATASETS\_IN\_MEMORY\_MAX\_SIZE=0 PYTHONPATH="src" deepspeed --num\_gpus=1 \\
> examples/research\_projects/wav2vec2/run\_common\_voice.py \\
> \--model\_name\_or\_path="facebook/wav2vec2-large-xlsr-53" --dataset\_config\_name="tr" \\
> \--output\_dir=./wav2vec2-large-xlsr-turkish-demo --overwrite\_output\_dir --num\_train\_epochs="5" \\
> \--per\_device\_train\_batch\_size="16" --learning\_rate="3e-4" --warmup\_steps="500" \\
> \--evaluation\_strategy="steps" --save\_steps="5" --eval\_steps="5" --logging\_steps="5" \\
> \--save\_total\_limit="3" --freeze\_feature\_extractor --feat\_proj\_dropout="0.0" --layerdrop="0.1" \\
> \--gradient\_checkpointing --group\_by\_length --do\_train --do\_eval --deepspeed \\
> examples/research\_projects/wav2vec2/ds\_config\_wav2vec2\_zero2.json 
> \`\`\`
> 
> Thanks to @patrickvonplaten for making small wav2vec2 models which helped a ton to debug faster and they were needed for the tests.
> 
> \## Requirements to merge this PR
> 
> \- \[x\] https://github.com/microsoft/DeepSpeed/pull/1135
> \- \[x\] deepspeed version requirement bumped to 0.4.0
> \- \[x\] deepspeed 0.4.0 released
> 
> Fixes: https://github.com/huggingface/transformers/issues/11446

wav2vec2 has 2 peculiarities:

1. it randomly skips layers! which I think is what requires `find_unused_parameters` - in normal dist and also in zero-2. for zero-3 we must run all gpus in sync, so this problem is removed. (see PR)
2. it uses `weight_norm` which re-creates 2 params in pre-forward which also has all kinds of potential side-effects. I am attempting to write a fused version of `weight_norm` + `Conv1d` which doesn’t use any tricks, but I haven’t fully sorted it out yet.

here is a work in progress:

```auto
import torch.nn as nn
from torch.nn.parameter import Parameter
from torch import _weight_norm, norm_except_dim
class Conv1dWithWeightNorm(nn.Conv1d):
    def __init__ (self, *args, **kwargs):
        super(Conv1dWithWeightNorm, self). __init__ (*args, **kwargs)
        self.dim = 2
        import deepspeed
        with deepspeed.zero.GatheredParameters(self.weight):
            weight = self.weight
        self.weight_g = Parameter(norm_except_dim(weight, 2, self.dim).data)
        self.weight_v = Parameter(weight.data)
        del self._parameters["weight"]
        self.weight = _weight_norm(self.weight_v, self.weight_g, self.dim)
        print(self.weight)

    def compute_weight(self):
        self.weight_g = Parameter(norm_except_dim(self.weight, 2, self.dim).data)
        self.weight_v = Parameter(self.weight.data)
        return _weight_norm(self.weight_v, self.weight_g, self.dim)

    def forward(self, input):
        self.weight = self.compute_weight()
        return self._conv_forward(input, self.weight, self.bias)

class Wav2Vec2PositionalConvEmbedding(nn.Module):
    def __init__ (self, config):
        super(). __init__ ()
        self.conv = Conv1dWithWeightNorm(
            in_channels=config.hidden_size,
            out_channels=config.hidden_size,
            kernel_size=config.num_conv_pos_embeddings,
            padding=config.num_conv_pos_embeddings // 2,
            groups=config.num_conv_pos_embedding_groups,
        )
        self.padding = Wav2Vec2SamePadLayer(config.num_conv_pos_embeddings)
        self.activation = ACT2FN[config.feat_extract_activation]

    def forward(self, hidden_states):
        hidden_states = hidden_states.transpose(1, 2)

        hidden_states = self.conv(hidden_states)
        hidden_states = self.padding(hidden_states)
        hidden_states = self.activation(hidden_states)

        hidden_states = hidden_states.transpose(1, 2)
        return hidden_states

```

---

<div class="post-metadata">

**Author:** ![DAlorlicorn](https://avatars.discourse-cdn.com/v4/letter/d/c4cdca/32.png) [@DAlorlicorn](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDAlorlicorn)%5C
**Post date:** [May 22, 2021, 11:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-with-multigpu%2F4894%2F18 "2021-05-22T23:47:57Z")

</div>

While it is a late reply, I also encountered the same situation as @liaspas. The culprit is actually the [gradient checkpointing](https://github.com/huggingface/transformers/blob/7eee950ac3135476c811daf23eca8cedbbaa3879/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L552) in the encoder. To be specific, it is an operation that reruns some forward pass instead of caching them during backward pass to save GPU memory usage. As the error message suggested, this will cause the torch ddp algorithm to mark them as ready twice. To resolve this, just disable it manually by setting gradient\_checkpointing to false for the encoder.config. Since gradient checkpointing exchange computation for memory, this will actually speed up the training given you have large enough VRAM. Also you need the find\_unused\_parameters=True as not all parameters are used in the forward pass.