# Errors running Inference Endpoint with quantized model
**URL:**
**Author:**  [@rfroli](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frfroli)%5C
**Post date:** [September 11, 2023, 10:43am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferrors-running-inference-endpoint-with-quantized-model%2F54444%2F1 "2023-09-11T10:43:00Z")
I’m encountering an issue while trying to load a quantized model into an Inference Endpoint. The model in question is “TheBloke/Llama-2-70B-chat-GPTQ”, and I’m using an AWS CPU Medium setup with PyTorch.
The error log suggests that the issue may be related to an incompatible version of `bitsandbytes`. Here’s the log for reference:
Error log :
2023/09/11 11:06:43 ~ Detected the presence of a `quantization_config` attribute in the model’s configuration but you don’t have the correct `bitsandbytes` version to support int8 serialization. Please install the latest version of `bitsandbytes` with `pip install --upgrade bitsandbytes`.
2023/09/11 11:06:51 ~ entrypoint.sh: line 13: 7 Killed uvicorn webservice\_starlette:app --host 0.0.0.0 --port 5000
Does anyone have any suggestions on how to resolve this? The goal is to deploy lower-cost, CPU-based inference endpoints in production.
---
**Author:**  [@Hatman](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHatman)%5C
**Post date:** [September 12, 2023, 10:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferrors-running-inference-endpoint-with-quantized-model%2F54444%2F2 "2023-09-12T22:17:15Z")
Can you throw your .py into a container?
---
**Author:**  [@rfroli](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frfroli)%5C
**Post date:** [September 14, 2023, 7:23am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferrors-running-inference-endpoint-with-quantized-model%2F54444%2F3 "2023-09-14T07:23:24Z")
I’m just getting started in this area, I’m unsure how to retrieve the .py file, since the Inference Endpoint is generated automatically upon model import…