# How to deploy quantized Mixtral 8x7b from Sagemaker?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-deploy-quantized-mixtral-8x7b-from-sagemaker%2F66541%3E%5C
**Category:** Amazon SageMaker\
**Created:** [December 21, 2023, 1:08am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-deploy-quantized-mixtral-8x7b-from-sagemaker%2F66541 "2023-12-21T01:08:13Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Soraheart1988](https://avatars.discourse-cdn.com/v4/letter/s/e9a140/32.png) [@Soraheart1988](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSoraheart1988)%5C
**Post date:** [December 21, 2023, 1:08am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-deploy-quantized-mixtral-8x7b-from-sagemaker%2F66541%2F1 "2023-12-21T01:08:13Z")

</div>

Please advise how to deploy 4bits or 8bits Mixtral 8x7 Model in sagemaker.  
I tried the following code but the deployment progress bar doesnt show up.

> model\_data = ‘s3://path/to/model.tar.gz’
> 
> pytorch\_model = PyTorchModel(  
> model\_data=model\_data,  
> role=role,  
> source\_dir =‘code’,  
> framework\_version=“1.12.1”,  
> entry\_point=“inference.py”,  
> py\_version =‘py38’,  
> model\_server\_workers=1  
> )
> 
> predictor = pytorch\_model.deploy(initial\_instance\_count=1,  
> instance\_type=“ml.g4dn.12xlarge”,  
> endpoint\_name = ‘test-mistral8x7b’,  
> serializer=CSVSerializer(),  
> deserializer=JSONDeserializer()  
> )

the code folder contains inference.py & requirements.txt

This is the inference code

> from transformers import AutoModelForCausalLM, AutoTokenizer
> 
> def model\_fn(model\_dir):
> 
> ```
> tokenizer = AutoTokenizer.from_pretrained(model_dir)
> model = AutoModelForCausalLM.from_pretrained(model_dir, load_in_4bit=True)
> return model, tokenizer
> 
> ```
> 
> def predict\_fn(data, model\_and\_tokenizer):
> 
> ```
> text = data.pop("text", data)
> # unpack model and tokenizer
> model, tokenizer = model_and_tokenizer
> inputs = tokenizer(text, return_tensors="pt").to(0)
> outputs = model.generate(**inputs, max_new_tokens=20)
> return tokenizer.decode(outputs[0], skip_special_tokens=True)
> 
> ```

The model.tar.tz is downloaded from and uploaded to the s3

> **[mistralai/Mixtral-8x7B-v0.1 at main](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1/tree/main)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.