# Deploying Mixtral8x7B on AWS Sagemaker from S3
**URL:**
**Author:**  [@PhysicsBeatsTech](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPhysicsBeatsTech)%5C
**Post date:** [May 24, 2024, 4:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdeploying-mixtral8x7b-on-aws-sagemaker-from-s3%2F88305%2F1 "2024-05-24T16:06:08Z")
I am trying to deploy the full-precision Mixtral 8x7B on AWS Sagemaker using the sagemaker-huggingface-inference-toolkit. I downloaded the model from the hub and compressed the modelfiles into a model.tar.gz file with all files residing directly in the root of the archive. I want to use the default code for inference and no custom inference.py.
I use the following code:
```auto
import json
import boto3
import sagemaker
from sagemaker.huggingface import HuggingFaceModel
role = sagemaker.get_execution_role() # execution role for the endpoint
sess = sagemaker.Session(default_bucket="some-bucket")
instance_type = "ml.g5.48xlarge"
number_of_gpu = 8
health_check_timeout = 300
s3_model_uri = f"s3://some_prefix_for_model/mistralai/Mixtral-8x7B-Instruct-v0.1/model.tar.gz"
llm_image = '763104351884.dkr.ecr.eu-central-1.amazonaws.com/huggingface-pytorch-tgi-inference:2.1.1-tgi2.0.1-gpu-py310-cu121-ubuntu22.04'
config = {
'HF_MODEL_ID': "/opt/ml/model",
'SM_NUM_GPUS': json.dumps(number_of_gpu),
'MAX_INPUT_LENGTH': json.dumps(24000),
'MAX_BATCH_PREFILL_TOKENS': json.dumps(32000),
'MAX_TOTAL_TOKENS': json.dumps(32000),
'MAX_BATCH_TOTAL_TOKENS': json.dumps(512000),
}
model = HuggingFaceModel(
role=role,
image_uri=llm_image,
env=config,
model_data=s3_model_uri,
)
llm = model.deploy(
initial_instance_count=1,
instance_type=instance_type,
container_startup_health_check_timeout=health_check_timeout,
)
```
The error I am getting is :
```auto
Error hosting endpoint huggingface-pytorch-tgi-inference-2024-05-24-15-14-56-313: Failed. Reason: Request to service failed. If failure persists after retry, contact customer support.. Try changing the instance type or reference the troubleshooting page https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference-troubleshooting.html
```
No cloudwatch logs are written unfortunately, so I don’t know exactly what’s going wrong. When I remove the model\_data dir and load the model directly from the Hub it works. This is why I am assuming that the decompression might be taking too long for the inference container to start. However, I do not know whether HugginfaceModel supports already decompressed model files, a question that was already asked [here](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-it-necessary-to-create-model-in-model-tar-gz-format-for-deployment-over-amazon-sagemaker%2F48964%2F2)%2C but not answered.
Thanks for any advice on how to move on here! 🙂
---
**Author:**  [@PhysicsBeatsTech](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPhysicsBeatsTech)%5C
**Post date:** [May 27, 2024, 10:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdeploying-mixtral8x7b-on-aws-sagemaker-from-s3%2F88305%2F2 "2024-05-27T10:11:36Z")
Update: Using local versions is working fine with the smaller & recently released Mistral 7B v3 on ml.g5.{12x,24x,48x}large machines. This strengthens my belief that it has sth. do to with the size of the model. Would be grateful for any advise!
---
**Author:**  [@PhysicsBeatsTech](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPhysicsBeatsTech)%5C
**Post date:** [June 11, 2024, 3:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdeploying-mixtral8x7b-on-aws-sagemaker-from-s3%2F88305%2F3 "2024-06-11T15:37:39Z")
I was able to fix it by passing
```auto
model_data_download_timeout=3600
```
as an argument to model.deploy(…)