# Inference error for FLAN-UL2 on AWS SageMaker **URL:** **Author:** ![Jorgeutd](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jorgeutd/32/39292_2.png) [@Jorgeutd](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJorgeutd)%5C **Post date:** [April 3, 2023, 3:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-error-for-flan-ul2-on-aws-sagemaker%2F35418%2F1 "2023-04-03T15:25:28Z") Following up on this thread any idea on this issue: ModelError Traceback (most recent call last) Cell In[12], line 17 7 parameters = { 8 “early\_stopping”: True, 9 “length\_penalty”: 2.0, (…) 13 “no\_repeat\_ngram\_size”: 3, 14 } 16 # Run prediction —\> 17 predictor.predict({ 18 “inputs”: payload, 19 “parameters” :parameters 20 }) 21 # [{‘generated\_text’: ‘Peter stayed with Elizabeth at the hospital for 3 days.’}] File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/predictor.py:161, in Predictor.predict(self, data, initial\_args, target\_model, target\_variant, inference\_id) 131 “”“Return the inference from the specified endpoint. 132 133 Args: (…) 155 as is. 156 “”” 158 request\_args = self.\_create\_request\_args( 159 data, initial\_args, target\_model, target\_variant, inference\_id 160 ) → 161 response = self.sagemaker\_session.sagemaker\_runtime\_client.invoke\_endpoint(\*\*request\_args) 162 return self.\_handle\_response(response) File ~/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py:530, in ClientCreator.\_create\_api\_method…\_api\_call(self, \*args, \*\*kwargs) 526 raise TypeError( 527 f"{py\_operation\_name}() only accepts keyword arguments." 528 ) 529 # The “self” in this scope is referring to the BaseClient. → 530 return self.\_make\_api\_call(operation\_name, kwargs) File ~/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py:960, in BaseClient.\_make\_api\_call(self, operation\_name, api\_params) 958 error\_code = parsed\_response.get(“Error”, {}).get(“Code”) 959 error\_class = self.exceptions.from\_code(error\_code) → 960 raise error\_class(parsed\_response, operation\_name) 961 else: 962 return parsed\_response ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (0) from primary with message “Your invocation timed out while waiting for a response from container primary. Review the latency metrics for each container in Amazon CloudWatch, resolve the issue, and try again.”. See [https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/sagemaker/Endpoints/llm-flan-ul2-20b-fp16-qh-2023-04-03](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/sagemaker/Endpoints/llm-flan-ul2-20b-fp16-qh-2023-04-03) in account 197614225699 for more information. Please see some of the logs errors: 1680533644222,“2023-04-03T14:54:02,158 [INFO] pool-2-thread-3 ACCESS\_LOG - /169.254.178.2:37372 ““GET /ping HTTP/1.1"” 200 1”,AllTraffic/i-09e62a922fe9422b0 1680533644222,“2023-04-03T14:54:03,987 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Prediction error”,AllTraffic/i-09e62a922fe9422b0 1680533644222,“2023-04-03T14:54:03,987 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last):”,AllTraffic/i-09e62a922fe9422b0 1680533644222,“2023-04-03T14:54:03,987 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/sagemaker\_huggingface\_inference\_toolkit/handler\_service.py””, line 219, in handle",AllTraffic/i-09e62a922fe9422b0 1680533644222,“2023-04-03T14:54:03,988 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.initialize(context)”,AllTraffic/i-09e62a922fe9422b0 1680533644222,“2023-04-03T14:54:03,988 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/sagemaker\_huggingface\_inference\_toolkit/handler\_service.py”“, line 77, in initialize”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,988 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.model = self.load(self.model\_dir)”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,988 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/sagemaker\_huggingface\_inference\_toolkit/handler\_service.py”“, line 104, in load”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,988 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - hf\_pipeline = get\_pipeline(task=os.environ[”“HF\_TASK”“], model\_dir=model\_dir, device=self.device)”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,988 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/sagemaker\_huggingface\_inference\_toolkit/transformers\_utils.py”“, line 272, in get\_pipeline”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,988 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - hf\_pipeline = pipeline(task=task, model=model\_dir, device=device, \*\*kwargs)”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,988 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/transformers/pipelines/ **init**.py”“, line 903, in pipeline”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,988 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return pipeline\_class(model=model, framework=framework, task=task, \*\*kwargs)”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,988 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/transformers/pipelines/text2text\_generation.py”“, line 65, in **init** ”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - super(). **init** (\*args, \*\*kwargs)”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/transformers/pipelines/base.py”“, line 780, in **init** ”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.model = self.model.to(self.device)”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/transformers/modeling\_utils.py”“, line 1749, in to”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return super().to(\*args, \*\*kwargs)”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py”“, line 989, in to”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self.\_apply(convert)”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py”“, line 641, in \_apply”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - module.\_apply(fn)”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py”“, line 641, in \_apply”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - module.\_apply(fn)”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py”“, line 641, in \_apply”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - module.\_apply(fn)”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,989 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - [Previous line repeated 4 more times]”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py”“, line 664, in \_apply”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - param\_applied = fn(param)”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py”“, line 987, in convert”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return t.to(device, dtype if t.is\_floating\_point() or t.is\_complex() else None, non\_blocking)”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 1; 22.20 GiB total capacity; 21.24 GiB already allocated; 222.12 MiB free; 21.24 GiB reserved in total by PyTorch) If reserved memory is \>\> allocated memory try setting max\_split\_size\_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH\_CUDA\_ALLOC\_CONF”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - “,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - During handling of the above exception, another exception occurred:”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - “,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last):”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/mms/service.py””, line 108, in predict”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - ret = self.\_entry\_point(input\_batch, self.context)”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,990 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “”/opt/conda/lib/python3.9/site-packages/sagemaker\_huggingface\_inference\_toolkit/handler\_service.py””, line 243, in handle”,AllTraffic/i-09e62a922fe9422b0 1680533644223,“2023-04-03T14:54:03,991 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise PredictionException(str(e), 400)”,AllTraffic/i-09e62a922fe9422b0 1680533645477,“2023-04-03T14:54:03,991 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 1; 22.20 GiB total capacity; 21.24 GiB already allocated; 222.12 MiB free; 21.24 GiB reserved in total by PyTorch) If reserved memory is \>\> allocated memory try setting max\_split\_size\_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH\_CUDA\_ALLOC\_CONF : 400”,AllTraffic/i-09e62a922fe9422b0 1680533645477,“2023-04-03T14:54:05,401 [INFO] W-9000-model com.amazonaws.ml.mms.wlm.WorkerThread - Backend response time: 165228”,AllTraffic/i-09e62a922fe9422b0 1680533647231,“2023-04-03T14:54:05,402 [INFO] W-9000-model ACCESS\_LOG - /169.254.178.2:53412 ““POST /invocations HTTP/1.1”” 400 165231”,AllTraffic/i-09e62a922fe9422b0 ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fphilschmid)%5C **Post date:** [April 3, 2023, 6:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-error-for-flan-ul2-on-aws-sagemaker%2F35418%2F2 "2023-04-03T18:03:41Z")
see [Inference failed for FLAN-UL2(20B) on SageMaker - #4 by philschmid](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-failed-for-flan-ul2-20b-on-sagemaker%2F35359%2F4)