# Wav2vec fine tuning - loss suddenly increased

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-loss-suddenly-increased%2F25696%3E%5C
**Category:** 🤗Transformers\
**Created:** [November 7, 2022, 6:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-loss-suddenly-increased%2F25696 "2022-11-07T18:44:03Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![DinBav](https://avatars.discourse-cdn.com/v4/letter/d/41988e/32.png) [@DinBav](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDinBav)%5C
**Post date:** [November 7, 2022, 6:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-loss-suddenly-increased%2F25696%2F1 "2022-11-07T18:44:03Z")

</div>

Hi,  
Experiencing sudden increase in loss when fine-tuning Wav2vec (see picture attached).  
We’ve seen a similar increase in loss due to the quality of data in the past (e.g. multiple white spaces between words or inaccurate transcription) - but this doesn’t seem to be the issue now.  
If the data is fine - what can cause this?  
Thank you 🙂

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/e/e10c2024ecd37da942b198365f749f63f350f732.jpeg)

---

<div class="post-metadata">

**Author:** ![sanchit-gandhi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sanchit-gandhi/32/21280_2.png) [@sanchit-gandhi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsanchit-gandhi)%5C
**Post date:** [November 8, 2022, 10:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwav2vec-fine-tuning-loss-suddenly-increased%2F25696%2F2 "2022-11-08T10:07:55Z")

</div>

I’ve experienced similar loss spikes in the past - they’re usually attributed to exploding gradients, which as you say can be traced back to the quality/noisiness of the training data. I typically try and compensate by adjusting the learning rate schedule, for which you have two dials to play with: the learning rate and the warm-up steps.

I’ve found training to be much more stable with CTC training by increasing the number of warm-up steps: since the learning rate ramps at a lower rate, you push the activations away from the pre-trained range much more slowly, and usually that results in more stable training. I’ve found 5000 warm-up steps to be a good value when training for 15000 total steps or more. In my experience, changing the warm-up steps has been more effective than lowering the learning rate!