# Inference failed for FLAN-UL2(20B) on SageMaker
**URL:**
**Author:**  [@akalitaa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fakalitaa)%5C
**Post date:** [April 2, 2023, 11:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-failed-for-flan-ul2-20b-on-sagemaker%2F35359%2F1 "2023-04-02T23:52:40Z")
I am using this blog to deploy FLAN-UL2 on SageMaker ([Deploy FLAN-UL2 20B on Amazon SageMaker](https://www.philschmid.de/deploy-flan-ul2-sagemaker)). Everything works fine except the last 2 steps on inferencing. During inferencing (predictor.predict) it is giving me the below error:
ValueError: (“You need to define one of the following [‘audio-classification’, ‘automatic-speech-recognition’, ‘feature-extraction’, ‘text-classification’, ‘token-classification’, ‘question-answering’, ‘table-question-answering’, ‘visual-question-answering’, ‘document-question-answering’, ‘fill-mask’, ‘summarization’, ‘translation’, ‘text2text-generation’, ‘text-generation’, ‘zero-shot-classification’, ‘zero-shot-image-classification’, ‘conversational’, ‘image-classification’, ‘image-segmentation’, ‘image-to-text’, ‘object-detection’, ‘zero-shot-object-detection’, ‘depth-estimation’, ‘video-classification’] as env ‘HF\_TASK’.”, 403)
Did anyone manage to make it work on SageMaker endpoint? Could you please recommend a way to fix the error…
---
**Author:**  [@philschmid](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fphilschmid)%5C
**Post date:** [April 3, 2023, 7:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-failed-for-flan-ul2-20b-on-sagemaker%2F35359%2F2 "2023-04-03T07:34:29Z")
Have you made any modifications? Can you please share the whole logs? it seems that your `model.tar.gz` didn’t get build correctly. Where are you running the code?
---
**Author:**  [@Jorgeutd](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJorgeutd)%5C
**Post date:** [April 3, 2023, 2:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-failed-for-flan-ul2-20b-on-sagemaker%2F35359%2F3 "2023-04-03T14:56:45Z")
Following up on this thread any idea on this issue:
ModelError Traceback (most recent call last)
Cell In[12], line 17
7 parameters = {
8 “early\_stopping”: True,
9 “length\_penalty”: 2.0,
(…)
13 “no\_repeat\_ngram\_size”: 3,
14 }
16 # Run prediction
—\> 17 predictor.predict({
18 “inputs”: payload,
19 “parameters” :parameters
20 })
21 # [{‘generated\_text’: ‘Peter stayed with Elizabeth at the hospital for 3 days.’}]
File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/predictor.py:161, in Predictor.predict(self, data, initial\_args, target\_model, target\_variant, inference\_id)
131 “”“Return the inference from the specified endpoint.
132
133 Args:
(…)
155 as is.
156 “””
158 request\_args = self.\_create\_request\_args(
159 data, initial\_args, target\_model, target\_variant, inference\_id
160 )
→ 161 response = self.sagemaker\_session.sagemaker\_runtime\_client.invoke\_endpoint(\*\*request\_args)
162 return self.\_handle\_response(response)
File ~/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py:530, in ClientCreator.\_create\_api\_method..\_api\_call(self, \*args, \*\*kwargs)
526 raise TypeError(
527 f"{py\_operation\_name}() only accepts keyword arguments."
528 )
529 # The “self” in this scope is referring to the BaseClient.
→ 530 return self.\_make\_api\_call(operation\_name, kwargs)
File ~/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py:960, in BaseClient.\_make\_api\_call(self, operation\_name, api\_params)
958 error\_code = parsed\_response.get(“Error”, {}).get(“Code”)
959 error\_class = self.exceptions.from\_code(error\_code)
→ 960 raise error\_class(parsed\_response, operation\_name)
961 else:
962 return parsed\_response
ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (0) from primary with message “Your invocation timed out while waiting for a response from container primary. Review the latency metrics for each container in Amazon CloudWatch, resolve the issue, and try again.”. See [https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/sagemaker/Endpoints/llm-flan-ul2-20b-fp16-qh-2023-04-03](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/sagemaker/Endpoints/llm-flan-ul2-20b-fp16-qh-2023-04-03) in account 197614225699 for more information.
Please see some of the logs errors:
1680533644222,“2023-04-03T14:54:02,158 [INFO] pool-2-thread-3 ACCESS\_LOG - /169.254.178.2:37372 ““GET /ping HTTP/1.1"” 200 1”,AllTraffic/i-09e62a922fe9422b0
1680533644222,“2023-04-03T14:54:03,987 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Prediction error”,AllTraffic/i-09e62a922fe9422b0
1680533644222,“2023-04-03T14:54:03,987 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last):”,AllTraffic/i-09e62a922fe9422b0
1680533644222,“2023-04-03T14:54:03,987 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/sagemaker\_huggingface\_inference\_toolkit/handler\_service.py””, line 219, in handle",AllTraffic/i-09e62a922fe9422b0
1680533644222,“2023-04-03T14:54:03,988 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.initialize(context)”,AllTraffic/i-09e62a922fe9422b0
1680533644222,“2023-04-03T14:54:03,988 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/sagemaker\_huggingface\_inference\_toolkit/handler\_service.py”“, line 77, in initialize”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,988 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.model = self.load(self.model\_dir)”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,988 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/sagemaker\_huggingface\_inference\_toolkit/handler\_service.py”“, line 104, in load”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,988 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - hf\_pipeline = get\_pipeline(task=os.environ[”“HF\_TASK”“], model\_dir=model\_dir, device=self.device)”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,988 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/sagemaker\_huggingface\_inference\_toolkit/transformers\_utils.py”“, line 272, in get\_pipeline”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,988 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - hf\_pipeline = pipeline(task=task, model=model\_dir, device=device, \*\*kwargs)”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,988 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/transformers/pipelines/ **init**.py”“, line 903, in pipeline”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,988 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return pipeline\_class(model=model, framework=framework, task=task, \*\*kwargs)”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,988 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/transformers/pipelines/text2text\_generation.py”“, line 65, in **init** ”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - super(). **init** (\*args, \*\*kwargs)”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/transformers/pipelines/base.py”“, line 780, in **init** ”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.model = self.model.to(self.device)”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/transformers/modeling\_utils.py”“, line 1749, in to”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return super().to(\*args, \*\*kwargs)”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py”“, line 989, in to”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self.\_apply(convert)”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py”“, line 641, in \_apply”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - module.\_apply(fn)”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py”“, line 641, in \_apply”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - module.\_apply(fn)”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py”“, line 641, in \_apply”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - module.\_apply(fn)”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - [Previous line repeated 4 more times]”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py”“, line 664, in \_apply”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - param\_applied = fn(param)”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py”“, line 987, in convert”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return t.to(device, dtype if t.is\_floating\_point() or t.is\_complex() else None, non\_blocking)”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 1; 22.20 GiB total capacity; 21.24 GiB already allocated; 222.12 MiB free; 21.24 GiB reserved in total by PyTorch) If reserved memory is \>\> allocated memory try setting max\_split\_size\_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH\_CUDA\_ALLOC\_CONF”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - “,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - During handling of the above exception, another exception occurred:”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - “,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last):”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/mms/service.py””, line 108, in predict”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - ret = self.\_entry\_point(input\_batch, self.context)”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/sagemaker\_huggingface\_inference\_toolkit/handler\_service.py””, line 243, in handle”,AllTraffic/i-09e62a922fe9422b0
1680533644223,“2023-04-03T14:54:03,991 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise PredictionException(str(e), 400)”,AllTraffic/i-09e62a922fe9422b0
1680533645477,“2023-04-03T14:54:03,991 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 1; 22.20 GiB total capacity; 21.24 GiB already allocated; 222.12 MiB free; 21.24 GiB reserved in total by PyTorch) If reserved memory is \>\> allocated memory try setting max\_split\_size\_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH\_CUDA\_ALLOC\_CONF : 400”,AllTraffic/i-09e62a922fe9422b0
1680533645477,“2023-04-03T14:54:05,401 [INFO] W-9000-model com.amazonaws.ml.mms.wlm.WorkerThread - Backend response time: 165228”,AllTraffic/i-09e62a922fe9422b0
1680533647231,"2023-04-03T14:54:05,402 [INFO] W-9000-model ACCESS\_LOG - /169.254.178.2:53412 ““POST /invocations HTTP/1.1"” 400 165231”,AllTraffic/i-09e62a922fe9422b0
---
**Author:**  [@philschmid](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fphilschmid)%5C
**Post date:** [April 3, 2023, 6:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-failed-for-flan-ul2-20b-on-sagemaker%2F35359%2F4 "2023-04-03T18:03:10Z")
@Jorgeutd could you share how you deployed the model with which versions?
---
**Author:**  [@akalitaa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fakalitaa)%5C
**Post date:** [April 3, 2023, 11:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-failed-for-flan-ul2-20b-on-sagemaker%2F35359%2F5 "2023-04-03T23:41:48Z")
@philschmid, I relooked at the model tar file, and there was an issue in terms of using pigz library. after fixing it, the inference in the notebook ran correctly. I was also able to use boto3 to invoke the endpoint. Superb!
---
**Author:**  [@Jorgeutd](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJorgeutd)%5C
**Post date:** [April 4, 2023, 2:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-failed-for-flan-ul2-20b-on-sagemaker%2F35359%2F6 "2023-04-04T14:10:33Z")
Hey @philschmid, I followed step by step your notebook; here is the deployment code:
from sagemaker.huggingface.model import HuggingFaceModel
# create Hugging Face Model Class
huggingface\_model = HuggingFaceModel(
model\_data=s3\_location, # path to your model and script
role=role, # iam role with permissions to create an Endpoint
transformers\_version=“4.26”, # transformers version used
pytorch\_version=“1.13”, # pytorch version used
py\_version=‘py39’, # python version used
model\_server\_workers=1,
name=model\_name
)
# deploy the endpoint endpoint
predictor = huggingface\_model.deploy(
initial\_instance\_count=1,
instance\_type=“ml.g5.12xlarge”,
endpoint\_name=endpoint\_name,
# container\_startup\_health\_check\_timeout=600, # increase timeout for large models
# model\_data\_download\_timeout=600 # increase timeout for large models
Here is the req file:
%%writefile code/requirements.txt
accelerate==0.16.0
transformers==4.26.0
bitsandbytes==0.37.0
---
**Author:**  [@Jorgeutd](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJorgeutd)%5C
**Post date:** [April 4, 2023, 3:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-failed-for-flan-ul2-20b-on-sagemaker%2F35359%2F7 "2023-04-04T15:45:09Z")
@philschmid I redeployed the model with the new changes on the notebook and now it is working fine.
%%writefile code/requirements.txt
accelerate==0.15.0
transformers==4.27.2
I also changed this part of the code:
from pathlib import Path
import os
# set HF\_HUB\_ENABLE\_HF\_TRANSFER env var to enable hf-transfer for faster downloads
os.environ[“HF\_HUB\_ENABLE\_HF\_TRANSFER”] = “1”
from huggingface\_hub import snapshot\_download
HF\_MODEL\_ID=“google/flan-ul2”
# Create a directory for the model in the specified directory
base\_dir = Path(“/home/ec2-user/SageMaker/”)
model\_tar\_dir = base\_dir.joinpath(“model\_directory”)
model\_tar\_dir.mkdir(exist\_ok=True)
# Set cache directory to be in the same filesystem as model\_tar\_dir
cache\_dir = base\_dir.joinpath(“.cache”)
cache\_dir.mkdir(exist\_ok=True)
# Download model from Hugging Face into model\_dir
snapshot\_download(HF\_MODEL\_ID, local\_dir=str(model\_tar\_dir), local\_dir\_use\_symlinks=False, cache\_dir=str(cache\_dir))
# copy code/ to model dir
copy\_tree(“code/”, str(model\_tar\_dir.joinpath(“code”)))
Is there an end to end example on how to fine tune a model like this?
Thanks,
Jorge