# How to parallelize inference on a quantized model
**URL:**
**Author:**  [@pvelosipednikov](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpvelosipednikov)%5C
**Post date:** [September 24, 2024, 8:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-parallelize-inference-on-a-quantized-model%2F108499%2F1 "2024-09-24T20:06:17Z")
I would like to parallelize generation across GPUs, but also load the model quantized.
The code below achieves task 1. How would I also incorporate loading the model in a quantized manner?
```auto
from transformers import pipeline, T5ForConditionalGeneration, AutoTokenizer, BitsAndBytesConfig
from accelerate import init_empty_weights, load_checkpoint_and_dispatch
checkpoint = 'google/flan-ul2'
with init_empty_weights():
model_loaded = T5ForConditionalGeneration.from_pretrained(checkpoint)
model_comb = load_checkpoint_and_dispatch(
model_loaded,
device_map='cuda',
no_split_module_classes=["T5Block"]
)
tokenizer = AutoTokenizer.from_pretrained(model)
generator = pipeline('text2text-generation', model=model_comb, tokenizer = tokenizer)
```
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [September 24, 2024, 10:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-parallelize-inference-on-a-quantized-model%2F108499%2F2 "2024-09-24T22:04:10Z")
Offloading to CPU may or may not be possible depending on the type of quantization library, but it seems to be possible for multi-GPU use, but I’m not sure if just specifying device\_map with the accelerate library will work.
If it’s still unsupported or buggy, I guess it would be like piecing together the following information to deal with it…
> **[Bitsandbytes](https://huggingface.co/docs/transformers/main/quantization/bitsandbytes)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
> **[Accelerated inference on NVIDIA GPUs](https://huggingface.co/docs/optimum-onnx/onnxruntime/usage_guides/gpu)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
> **[Parallelism methods](https://huggingface.co/docs/transformers/perf_train_gpu_many)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
> **[Quantization-Aware Training for Large Language Models with PyTorch – PyTorch](https://pytorch.org/blog/quantization-aware-training/)**
But well, I don’t know how people have multiple GPUs and such.
I’ve seen at least two people on the forum complain that **load balancing to multiple GPUs doesn’t work properly** , so you’re on the hook for that. That might be a bug.
> [@CUDA Memory issue for model.generate() in AutoModelForCausalLM](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-memory-issue-for-model-generate-in-automodelforcausallm%2F74216%2F1)%3A
>
> Hello everyone, I have 4 A100 GPUs and I’m utilizing Mixtral with dtype set as bfloat16 for a text generation task on these GPUs. I’m aware that by using device\_map="balanced\_low\_0", I can distribute the model across GPUs 1, 2, and 3, while leaving GPU 0 available for the model.generate() function, as detailed in the documentation here: [[Handling big models for inference](https://huggingface.co/docs/accelerate/en/concept_guides/big_model_inference)]. The Mixtral model with bfloat16 type occupies approximately 90 GB, which is loaded onto GPUs 1, 2, and partially on GPU 3.…
---
**Author:**  [@pvelosipednikov](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpvelosipednikov)%5C
**Post date:** [September 24, 2024, 10:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-parallelize-inference-on-a-quantized-model%2F108499%2F3 "2024-09-24T22:49:59Z")
Thanks for the response @John6666 . So I figured out from [this](https://huggingface.co/docs/accelerate/usage_guides/big_modeling) post that just using the regular syntax with ‘auto’ does parallelize GPUs automatically. The relevant part is at the bottom of the page.
I’m still new at that, so interestingly enough out of the 4 GPUs I have, 1 of them is not getting any utilization at all.
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [September 24, 2024, 11:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-parallelize-inference-on-a-quantized-model%2F108499%2F4 "2024-09-24T23:08:28Z")
The HF manual consists of its introduction and a type that is automatically generated from the library. (They extract what’s written as comments in the code.)
The introduction often contains theoretical ideals and information that was correct at the time it was written but is now incorrect, so ultimately it is quicker to read the library code or watch and steal the work of others who have done it well.
It would be easiest if it could be fixed by updating the library…
But when it comes to multi-GPUs, few people use them on HF’s Spaces, so if it’s buggy, you’ll have to do it manually with torch.
[https://pytorch.org/tutorials/intermediate/model\_parallel\_tutorial.html](https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html)
---
**Author:**  [@pvelosipednikov](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpvelosipednikov)%5C
**Post date:** [October 6, 2024, 11:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-parallelize-inference-on-a-quantized-model%2F108499%2F5 "2024-10-06T23:48:10Z")
Thanks. Is the transformers implementation buggy though? I can’t tell one way or the other.
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [October 7, 2024, 12:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-parallelize-inference-on-a-quantized-model%2F108499%2F6 "2024-10-07T00:13:58Z")
Transformers **itself** have a long history and there are not many bugs in their implementation. I’m not saying there aren’t.
The problem is when they are combined with libraries for quantization and accelerate.
accelerate is doing advanced things, and therefore bugs are likely to occur.
The quantization libraries are still being updated frequently, so they are bug prone.
The fastest workaround on the user side is to try using a different library for quantization?
The next best thing is to find versions of the libraries that works well and fix them. The last resort is to go all manual with torch.
Recently, **torchao, the official torch quantization** , has been released, so you might try this.
> **[GitHub - sayakpaul/diffusers-torchao: End-to-end recipes for optimizing diffusion models...](https://github.com/sayakpaul/diffusers-torchao)**
>
> End-to-end recipes for optimizing diffusion models with torchao and diffusers (inference and FP8 training).