# Databricks models deployment to sagemaker are not working
**URL:**
**Author:**  [@radsimu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fradsimu)%5C
**Post date:** [April 18, 2023, 11:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatabricks-models-deployment-to-sagemaker-are-not-working%2F37034%2F1 "2023-04-18T23:23:18Z")
For the databricks models, none of the examples for deploying on Amazon SageMaker work. The suggested parameters for deploying as a Sagemaker endpoint are not working. I mean the endpoint deployment works but when trying to call the endpoint, it fails. The suggested parameters for instantiating the sagemaker. HuggingFaceModel seem off, and the instance type is deffinitely too weak.
After adjusting the suggested deployment configs to ‘transformers\_version’: ‘4.26.0’, ‘pytorch\_version’: ‘1.13.1’, and a more powerful endpoint type, now I am told that I need to set the option trust\_remote\_code=True when the pipeline is called from inside the docker that was deployed on the endpoint. How can I determine that parameter to be passed as true? If that is not doable, then none of the dolly models are usable as out of the box sagemaker deplyments
---
**Author:**  [@philschmid](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fphilschmid)%5C
**Post date:** [April 19, 2023, 7:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatabricks-models-deployment-to-sagemaker-are-not-working%2F37034%2F2 "2023-04-19T07:27:05Z")
Can you please share the code you used to deploy the model you are talking about? Hard to understand your issue without any details
---
**Author:**  [@radsimu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fradsimu)%5C
**Post date:** [April 19, 2023, 4:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatabricks-models-deployment-to-sagemaker-are-not-working%2F37034%2F3 "2023-04-19T16:05:14Z")
the code is already there, it is suggested on the huggingface site. Go to any of the databricks models (ex [databricks/dolly-v2-12b · Hugging Face](https://huggingface.co/databricks/dolly-v2-12b)) then in the upper right there is a Deploy button dropdown, click on that, select Amazon SageMaker (the only option actually), then choose pretty much anything (I tried: text generation on AWS). Try to run that code as is. Model will be deployed, but then it fails on predict:
cloudwatch logs:
> com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.8/site-packages/transformers/models/auto/configuration\_auto.py”, line 372, in **getitem**
> com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise KeyError(key)
> [INFO] W-databricks\_\_dolly-v2-3b-4-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - KeyError: ‘gpt\_neox’
Then, fiddling with the suggested parameters seems to fix the gpt\_neox keyError issue. These are the adjustments I made:
```auto
huggingface_model = HuggingFaceModel(
transformers_version='4.26.0',
pytorch_version='1.13.1',
py_version='py39',
env=hub,
role=role,
)
```
But brings us to the following error (I also tried with a ml.g5.8xlarge instance type - same error):
> com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.9/site-packages/transformers/pipelines/ **init**.py”, line 704, in pipeline
> com.amazonaws.ml.mms.wlm.WorkerLifeCycle - ValueError: Loading this pipeline requires you to execute the code in the pipeline file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option `trust_remote_code=True` to remove this error.
This has to do with the way the pipeline is called from within the sagemaker docker image. For some reason, it is considered a “custom” pipeline and one has to explicitly set `trust_remote_code=True` when calling it. How can I get the already built and published docker image to pass `trust_remote_code=True` when calling the internal prediction pipeline? Or maybe I am doing something else wrong and that is not at all necessary.
---
**Author:**  [@mcapizzi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmcapizzi)%5C
**Post date:** [May 23, 2023, 11:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatabricks-models-deployment-to-sagemaker-are-not-working%2F37034%2F4 "2023-05-23T23:00:14Z")
Anybody figure this one out?
---
**Author:**  [@mcapizzi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmcapizzi)%5C
**Post date:** [May 23, 2023, 11:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatabricks-models-deployment-to-sagemaker-are-not-working%2F37034%2F5 "2023-05-23T23:03:56Z")
As expected, it looks like it gets a little hacky. But [here](https://huggingface.co/databricks/dolly-v2-12b/discussions/44#6442c3028569978432faba7e) is a solution.
---
**Author:**  [@philschmid](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fphilschmid)%5C
**Post date:** [May 24, 2023, 6:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatabricks-models-deployment-to-sagemaker-are-not-working%2F37034%2F6 "2023-05-24T06:24:05Z")
@mcapizzi the link you shared is the current best way. Since we are not having a way to tell the toolkit that you are okay with using “remote\_code” let me add that to our backlog.
---
**Author:**  [@mcapizzi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmcapizzi)%5C
**Post date:** [May 24, 2023, 2:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatabricks-models-deployment-to-sagemaker-are-not-working%2F37034%2F7 "2023-05-24T14:30:29Z")
Thanks @philschmid ! I’m sure it’s impossible to accommodate every model and type so I’d understand if the subset of models that require “remote code” is so small that it’s not a priority. But I _love_ the current functionality as it is! Got me up and running on some deployment tests in less than an hour.