# ValueError fp16 lm\_head.weight

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fvalueerror-fp16-lm-head-weight%2F10433%3E%5C
**Category:** DeepSpeed\
**Created:** [October 1, 2021, 12:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fvalueerror-fp16-lm-head-weight%2F10433 "2021-10-01T12:25:40Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![TallBoi](https://avatars.discourse-cdn.com/v4/letter/t/9dc877/32.png) [@TallBoi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTallBoi)%5C
**Post date:** [October 1, 2021, 12:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fvalueerror-fp16-lm-head-weight%2F10433%2F1 "2021-10-01T12:25:41Z")

</div>

I am trying to run [run\_translation.py](https://github.com/huggingface/transformers/tree/master/examples/pytorch/translation) with mt5-large and DeepSpeed enabled. I use [ds\_config\_zero3.json](https://github.com/huggingface/transformers/tree/0f221d2cce751182c455295ef2c03a2c1bd3d66b/tests/deepspeed) as the config file. However, when I try to run this, I get the following error:

> ValueError: fp16 is enabled but the following parameters have dtype that is not fp16: lm\_head.weight

Is there some config setting I’m missing that could help resolve this issue?

---

<div class="post-metadata">

**Author:** ![alexerdmann](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alexerdmann/32/2872_2.png) [@alexerdmann](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Falexerdmann)%5C
**Post date:** [October 24, 2021, 3:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fvalueerror-fp16-lm-head-weight%2F10433%2F2 "2021-10-24T15:58:29Z")

</div>

Hey did you figure out to resolve this? I’d be interested to learn what you did.

I ran the ASR example [here](https://github.com/huggingface/transformers/tree/master/examples/research_projects/wav2vec2), and it ran fine, but I noticed it had fp16 set to false. If I try to save memory by passing --fp16 at the command line or manually invoking fp16=True when calling TrainingArguments, I get the same error you report.