# Mistral AI Sagemaker deployment failing

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmistral-ai-sagemaker-deployment-failing%2F57379%3E%5C
**Category:** Amazon SageMaker\
**Created:** [October 4, 2023, 1:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmistral-ai-sagemaker-deployment-failing%2F57379 "2023-10-04T13:25:44Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mus-shd](https://avatars.discourse-cdn.com/v4/letter/m/67e7ee/32.png) [@mus-shd](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmus-shd)%5C
**Post date:** [October 4, 2023, 1:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmistral-ai-sagemaker-deployment-failing%2F57379%2F1 "2023-10-04T13:25:44Z")

</div>

Hi 👋  
Trying to deploy `mistralai/Mistral-7B-Instruct-v0.1` model on AWS Sagemaker using the Hugging Face LLM DLC. Here is my code:

```auto
import json
from sagemaker.huggingface import HuggingFaceModel
from sagemaker.huggingface import get_huggingface_llm_image_uri

# retrieve the llm image uri
llm_image = get_huggingface_llm_image_uri(
  "huggingface",
  version="1.0.3"
)

# sagemaker config
instance_type = "ml.g5.2xlarge"
number_of_gpu = 1
health_check_timeout = 600

# Define Model and Endpoint configuration parameter
config = {
  'HF_MODEL_ID': "mistralai/Mistral-7B-Instruct-v0.1",
  'SM_NUM_GPUS': json.dumps(number_of_gpu),
  'MAX_INPUT_LENGTH': json.dumps(2048),
  'MAX_TOTAL_TOKENS': json.dumps(4096),
  'MAX_BATCH_TOTAL_TOKENS': json.dumps(8192),
  'HUGGING_FACE_HUB_TOKEN': "hf_ ****"
}

# create HuggingFaceModel with the image uri
llm_model = HuggingFaceModel(
  role=role,
  image_uri=llm_image,
  env=config,
code_location=f"s3://{S3_BUCKET}/"
)

llm = llm_model.deploy(
    initial_instance_count=1,
    instance_type=instance_type,
    endpoint_name="mistral-7b-instruct",
    container_startup_health_check_timeout=health_check_timeout,
    tags=[{"Key": "ENV", "Value": "dev"}]
)

```

The deployment fails with the following errors:

```auto
Download encountered an error: Traceback (most recent call last):
  File "/opt/conda/bin/text-generation-server", line 8, in <module>
    sys.exit(app())
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/cli.py", line 195, in download_weights
    utils.convert_files(local_pt_files, local_st_files, discard_names)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/convert.py", line 106, in convert_files
    convert_file(pt_file, sf_file, discard_names)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/convert.py", line 68, in convert_file
    to_removes = _remove_duplicate_names(loaded, discard_names=discard_names)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/convert.py", line 32, in _remove_duplicate_names
    raise RuntimeError(

```

```auto
RuntimeError: Error while trying to find names to remove to save state dict, but found no suitable name to keep for saving amongst: {'model.norm.weight'}. None is covering the entire storage.Refusing to save/load the model since you could be storing much more memory than needed. Please refer to https://huggingface.co/docs/safetensors/torch_shared_tensors for more information. Or open an issue.

```

Any idea how to solve this issue and make the deployment work? Thanks in advance for your help 🙏

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fphilschmid)%5C
**Post date:** [October 4, 2023, 1:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmistral-ai-sagemaker-deployment-failing%2F57379%2F2 "2023-10-04T13:52:44Z")

</div>

Mistral is not supported in 1.0.3 since the model came afterwards, please try 1.1.0

---

<div class="post-metadata">

**Author:** ![mus-shd](https://avatars.discourse-cdn.com/v4/letter/m/67e7ee/32.png) [@mus-shd](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmus-shd)%5C
**Post date:** [October 4, 2023, 2:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmistral-ai-sagemaker-deployment-failing%2F57379%2F3 "2023-10-04T14:20:07Z")

</div>

Lot of thanks for the quick reply 🙏 Indeed, with version 1.1.0 it is working like a charm!

---

<div class="post-metadata">

**Author:** ![piyushaaryan011](https://avatars.discourse-cdn.com/v4/letter/p/67e7ee/32.png) [@piyushaaryan011](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpiyushaaryan011)%5C
**Post date:** [December 29, 2023, 8:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmistral-ai-sagemaker-deployment-failing%2F57379%2F4 "2023-12-29T20:51:02Z")

</div>

hey , i have trained zephyr-7b on qlora. now i am trying to deploy to the endpoint .  
but got this error:-

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/8/2/82484578ae6ebb3a5b581372f7dff02f31c90b88.png)  
it’s saying unsupported model type mistral , can you share what i can do to solve this.