# Why is Wav2Vec pretraining loss not decreasing?
**URL:**
**Author:**  [@ZaNi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FZaNi)%5C
**Post date:** [July 6, 2021, 7:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-wav2vec-pretraining-loss-not-decreasing%2F8112%2F1 "2021-07-06T19:34:27Z")
Hi there everyone 🤗
I’m currently trying to train a Wav2Vec base model. During the pre-training phase, The loss starts off around 4, decreases and then shoots up to 6.658 and stays there. The accuracy is also low and does not increase. My learning rate is set at 0.005. I started off with a learning rate of 0.0001 and started increasing it gradually when I saw these results. I use the english Wav2Vec model for weight initialisation. I thought it would improve if I waited longer but it stays the same even after 20 epochs. Can anyone please share some advice on what I could do to avoid this and improve the training?
Your assistance will be much appreciated! 🤗
---
**Author:**  [@KhusainovAidar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKhusainovAidar)%5C
**Post date:** [July 21, 2021, 7:39am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-wav2vec-pretraining-loss-not-decreasing%2F8112%2F2 "2021-07-21T07:39:07Z")
Hi ZaNi,
Have you found the reason for such behaviour? I’m facing the same problem pretraining my model from English base model.
---
**Author:**  [@AndySun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAndySun)%5C
**Post date:** [July 29, 2021, 1:45am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-wav2vec-pretraining-loss-not-decreasing%2F8112%2F3 "2021-07-29T01:45:50Z")
@ZaNi @KhusainovAidar Hi, I have the same problem. Is there any solution?
---
**Author:**  [@AndySun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAndySun)%5C
**Post date:** [July 29, 2021, 1:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-wav2vec-pretraining-loss-not-decreasing%2F8112%2F4 "2021-07-29T01:54:18Z")
Hi @patrickvonplaten
Thank you for your time!
I followed this pretrain script, but the loss is not decreasing. Like this
Could you give us some advice？ Thank you!
Andy
---
**Author:**  [@KhusainovAidar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKhusainovAidar)%5C
**Post date:** [July 29, 2021, 4:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-wav2vec-pretraining-loss-not-decreasing%2F8112%2F5 "2021-07-29T16:03:32Z")
@AndySun I haven’t found it. The strange thing that after restarting training from checkpoint (for which loss score was already near zero) it shows more realistic loss scores for first several steps and after that goes to 0.0003 again. So for now I’ve just trained wav2vec2 with fairseq and converted the resulting model into TorchScript.
---
**Author:**  [@AndySun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAndySun)%5C
**Post date:** [July 30, 2021, 12:36am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-wav2vec-pretraining-loss-not-decreasing%2F8112%2F6 "2021-07-30T00:36:46Z")
@KhusainovAidar Okay. Thanks for your information!
---
**Author:**  [@AndySun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAndySun)%5C
**Post date:** [July 31, 2021, 8:59am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-wav2vec-pretraining-loss-not-decreasing%2F8112%2F7 "2021-07-31T08:59:27Z")
@KhusainovAidar Hi Bro, could you tell us how to convert the fairseq pre-trained model to Transformers wav2vec formant?
Thank you!
---
**Author:**  [@KhusainovAidar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKhusainovAidar)%5C
**Post date:** [July 31, 2021, 9:36am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-wav2vec-pretraining-loss-not-decreasing%2F8112%2F8 "2021-07-31T09:36:19Z")
There is no such option, at least I haven’t found it. I’m using this repo to convert fairseq model to TorchScript and use it in ‘server’-mode [audio/examples/libtorchaudio/speech\_recognition at master · pytorch/audio · GitHub](https://github.com/pytorch/audio/tree/master/examples/libtorchaudio/speech_recognition)
---
**Author:**  [@AndySun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAndySun)%5C
**Post date:** [July 31, 2021, 9:51am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-wav2vec-pretraining-loss-not-decreasing%2F8112%2F9 "2021-07-31T09:51:17Z")
Okay! The TorchScript can be used for ASR product. Thanks a lot!
---
**Author:**  [@AndySun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAndySun)%5C
**Post date:** [August 10, 2021, 11:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-wav2vec-pretraining-loss-not-decreasing%2F8112%2F10 "2021-08-10T11:06:25Z")
Hi, @KhusainovAidar
I found one solution to convert fairseq pre-trained model to Transformers format. For your reference.
python -m transformers.models.wav2vec2.convert\_wav2vec2\_original\_pytorch\_checkpoint\_to\_pytorch --pytorch\_dump\_folder\_path ./converted\_model/ --checkpoint\_path /path/to/\*\*.pt --not\_finetuned
---
**Author:**  [@mfox](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmfox)%5C
**Post date:** [April 29, 2022, 4:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-wav2vec-pretraining-loss-not-decreasing%2F8112%2F11 "2022-04-29T16:53:55Z")
I’m also having this issue. Has anybody figured out what the issue is? I’m using the following hyperparameters:
```auto
python ./code/pretrain.py \
--dataset_name="./code/VoicesOfColor" \
--dataset_config_names="train" \
--dataset_split_names="TEST" \
--output_dir="./code/wav2vec2-pretrained-VOC/artefacts" \
--model_name_or_path="patrickvonplaten/wav2vec2-base-v2" \
--max_train_steps=600000 \
--num_warmup_steps=32000 \
--gradient_accumulation_steps=4 \
--learning_rate=0.001 \
--weight_decay=0.01 \
--max_duration_in_seconds=30.0 \
--min_duration_in_seconds=2.0 \
--logging_steps=1 \
--saving_steps=10000 \
--per_device_train_batch_size=8 \
--per_device_eval_batch_size=8 \
--adam_beta1=0.9 \
--adam_beta2=0.98 \
--adam_epsilon=1e-06
```