# Finetuning MT0 produce 0 loss

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetuning-mt0-produce-0-loss%2F52299%3E%5C
**Category:** Models\
**Created:** [August 26, 2023, 9:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetuning-mt0-produce-0-loss%2F52299 "2023-08-26T21:04:08Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![VicDon](https://avatars.discourse-cdn.com/v4/letter/v/c0e974/32.png) [@VicDon](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FVicDon)%5C
**Post date:** [August 26, 2023, 9:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetuning-mt0-produce-0-loss%2F52299%2F1 "2023-08-26T21:04:08Z")

</div>

I would like to finetune a MT0 model with fp16, int8 or int4. However, the loss is always 0 because of `nan`. Wondering how to fix this 0 loss issue ?

`[INFO|trainer.py:327] 2023-08-26 21:05:39,168 >> {'loss': 3.3959, 'learning_rate': 9.993190040434134e-07, 'train_runtime': 14.3111, 'train_samples_per_second': 8.944, 'train_num_samples_consumed': 128, 'job_progress': 0.0006809959565865078, 'epoch': 0.0} [INFO|trainer.py:327] 2023-08-26 21:05:52,255 >> {'loss': 0.0, 'learning_rate': 9.986380080868269e-07, 'train_runtime': 13.0882, 'train_samples_per_second': 9.78, 'train_num_samples_consumed': 256, 'job_progress': 0.0013619919131730156, 'epoch': 0.01} [INFO|trainer.py:327] 2023-08-26 21:06:05,317 >> {'loss': 0.0, 'learning_rate': 9.979570121302404e-07, 'train_runtime': 13.0619, 'train_samples_per_second': 9.799, 'train_num_samples_consumed': 384, 'job_progress': 0.002042987869759523, 'epoch': 0.01}`

---

<div class="post-metadata">

**Author:** ![mikojelly](https://avatars.discourse-cdn.com/v4/letter/m/3ab097/32.png) [@mikojelly](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmikojelly)%5C
**Post date:** [September 5, 2023, 8:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetuning-mt0-produce-0-loss%2F52299%2F2 "2023-09-05T20:00:02Z")

</div>

Hi,  
it seems like there is overflow happening due to fp16 being too narrow. Maybe set it to bf16 instead.

From mt0 model pages, I saw it is training with bf16

- **Finetuning steps:** 25000
- **Finetuning tokens:** 4.62 billion
- **Precision:** bfloat16