# Gettings nan with deepspeed

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgettings-nan-with-deepspeed%2F4786%3E%5C
**Category:** 🤗Transformers\
**Created:** [March 20, 2021, 11:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgettings-nan-with-deepspeed%2F4786 "2021-03-20T11:56:55Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dara](https://avatars.discourse-cdn.com/v4/letter/d/f14d63/32.png) [@Dara](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDara)%5C
**Post date:** [March 20, 2021, 11:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgettings-nan-with-deepspeed%2F4786%2F1 "2021-03-20T11:56:55Z")

</div>

Hi

I am adding adapter layers [1] between the layers of MT5 model, I am using deepspeed to run the models and I am always getting NaNs as training loss, I greatly appreciate any advice @stas on how one can make the training with deepspeed more stable and resolve the nan problem. Is there any parameters/setting I can play with when using deepspeed for better stability?

thank you very much

[1] [https://arxiv.org/pdf/1902.00751.pdf](https://arxiv.org/pdf/1902.00751.pdf)