# Enabling load\_in\_8bit makes inference much slower

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fenabling-load-in-8bit-makes-inference-much-slower%2F38596%3E%5C
**Category:** 🤗Transformers\
**Created:** [May 3, 2023, 9:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fenabling-load-in-8bit-makes-inference-much-slower%2F38596 "2023-05-03T21:36:35Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![feizheng](https://avatars.discourse-cdn.com/v4/letter/f/ed655f/32.png) [@feizheng](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffeizheng)%5C
**Post date:** [May 3, 2023, 9:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fenabling-load-in-8bit-makes-inference-much-slower%2F38596%2F1 "2023-05-03T21:36:35Z")

</div>

I loaded the 7b llama on an A100 this way:

```auto
quantization_config = transformers.BitsAndBytesConfig(load_in_8bit=False, llm_int8_threshold=0.0)
model = LlamaForCausalLM.from_pretrained(
   "decapoda-research/llama-7b-hf",
    device_map="auto",
    torch_dtype=torch.float16,
    quantization_config=quantization_config,
)

```

When `load_in_8bit` is `False`, it generates 16.7 tokens/sec whereas `load_in_8bit=True` generates only 6.7 tokens/sec. It seems I probably set up configurations incorrectly for `load_in_8bit=True`.

My transformers version is `4.29.0.dev0`. Did I miss anything? Thanks.

---

<div class="post-metadata">

**Author:** ![abhishek-wfx](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@abhishek-wfx](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabhishek-wfx)%5C
**Post date:** [August 30, 2023, 1:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fenabling-load-in-8bit-makes-inference-much-slower%2F38596%2F2 "2023-08-30T13:09:58Z")

</div>

I am facing similar issue, did you find a solution?

---

<div class="post-metadata">

**Author:** ![chaochaoli](https://avatars.discourse-cdn.com/v4/letter/c/9fc348/32.png) [@chaochaoli](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fchaochaoli)%5C
**Post date:** [September 9, 2023, 7:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fenabling-load-in-8bit-makes-inference-much-slower%2F38596%2F3 "2023-09-09T07:48:03Z")

</div>

me too，and training is slower too

---

<div class="post-metadata">

**Author:** ![AbhijitHerekar](https://avatars.discourse-cdn.com/v4/letter/a/eada6e/32.png) [@AbhijitHerekar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAbhijitHerekar)%5C
**Post date:** [February 13, 2024, 7:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fenabling-load-in-8bit-makes-inference-much-slower%2F38596%2F4 "2024-02-13T07:05:40Z")

</div>

same here, not sure if anyone found the root cause.