# Unable to load LLM with load\_in\_8bits

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-load-llm-with-load-in-8bits%2F39114%3E%5C
**Category:** 🤗Transformers\
**Created:** [May 8, 2023, 5:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-load-llm-with-load-in-8bits%2F39114 "2023-05-08T17:52:31Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![abhinavkulkarni](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/abhinavkulkarni/32/32750_2.png) [@abhinavkulkarni](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabhinavkulkarni)%5C
**Post date:** [May 8, 2023, 5:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-load-llm-with-load-in-8bits%2F39114%2F1 "2023-05-08T17:52:31Z")

</div>

Hi,

I am trying to load `mosaicml/mpt-7b-instruct` with `load_in_8bits=True` and `device_map="auto"`, however I get the following error:

```python
ValueError: MPTForCausalLM does not support `device_map='auto'` yet.

```

How can I get rid of this error?

Thanks!

---

<div class="post-metadata">

**Author:** ![abhinavkulkarni](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/abhinavkulkarni/32/32750_2.png) [@abhinavkulkarni](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabhinavkulkarni)%5C
**Post date:** [May 9, 2023, 6:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-load-llm-with-load-in-8bits%2F39114%2F2 "2023-05-09T06:19:22Z")

</div>

Thanks to @sam-mosaic, I got a reply on the model [community forum](https://huggingface.co/spaces/mosaicml/mpt-7b-instruct/discussions/3).

Are there any specific set of steps to add `int8` inference support for an LLM?

Thanks!