# Transformers 4.9.0 on SageMaker
**URL:**
**Author:**  [@dzorlu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdzorlu)%5C
**Post date:** [August 10, 2021, 10:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-4-9-0-on-sagemaker%2F9145%2F1 "2021-08-10T22:45:13Z")
Hello-
I am working on deploying a speech-recognition app using HuggingFace following the instructions [here](https://huggingface.co/docs/sagemaker/inference#overview). My understanding is that the inference toolkit uses `pipelines`, but the speech-recognition is only introduced with the \> `4.9.0` releases, whereas the current [AWS images](https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/image_uri_config/huggingface.json) are pointing to `4.6.x`.
Is there any way around this? What do you think suggest that I do to make the deployment work? My hunch is that I need to supply a new `image_uri`.
Thank you!
Deniz
.
---
**Author:**  [@philschmid](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fphilschmid)%5C
**Post date:** [August 11, 2021, 5:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-4-9-0-on-sagemaker%2F9145%2F2 "2021-08-11T05:55:33Z")
Hello @dzorlu,
Great to hear that you are working on a speech task!! Yes, the inference toolkit uses the `pipelines` from transformers. The code is open source you want to take a deeper look [GitHub - aws/sagemaker-huggingface-inference-toolkit](https://github.com/aws/sagemaker-huggingface-inference-toolkit).
I am happy to share that we are working on new releases for the DLC, which include `4.9` and higher. Sadly I think it will take around 2 more weeks to be around.
In the meantime, you could use the official DLC and provide as `model_data` a `model.tar.gz` which contains a custom module, documented here: [Deploy models to Amazon SageMaker](https://huggingface.co/docs/sagemaker/inference#user-defined-codemodules)
With a custom module, you can provide a `requirements.txt` to upgrade the dependencies and then provide a `inference.py` with a custom `model_fn` to load the `asr` pipeline.
---
**Author:**  [@dzorlu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdzorlu)%5C
**Post date:** [August 11, 2021, 11:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-4-9-0-on-sagemaker%2F9145%2F3 "2021-08-11T23:55:02Z")
Thank you Philipp for the quick response and the direction. I will try the `model_data` route. It’s also great to hear that you are working on updating the DLCs. The team’s work is always much appreciated!
Best
Deniz
---
**Author:**  [@parasmehta](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fparasmehta)%5C
**Post date:** [March 17, 2022, 4:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-4-9-0-on-sagemaker%2F9145%2F4 "2022-03-17T16:54:37Z")
Hi @philschmid, I am trying to exactly the same thing as @dzorlu, so I was pleasantly surprised to see that someone else asked this question a few months ago. I am trying to deploy a speech recognition model from the Hugging Face model hub for inference on a Sagemaker notebook using the Inference Toolkit using the instructions here [https://huggingface.co/docs/sagemaker/inference#deploy-a-model-from-the-%F0%9F%A4%97-hub](https://huggingface.co/docs/sagemaker/inference#deploy-a-model-from-the-%F0%9F%A4%97-hub). However, when I do `predictor.predict({inputs': "audio.wav"})`, I get the error `Unknown task automatic-speech-recognition ....` I guess this is because of the transformers version in the DLC?
```auto
huggingface_model = HuggingFaceModel(
transformers_version='4.6.1',
pytorch_version='1.7.1',
py_version='py36',
env=hub,
role=role,
)
```
If so, is there meanwhile a way to use a newer version than `4.6.1` on Sagemaker so that I can use the inference toolkit to deploy an ASR model from the hub? Thanks in advance!
---
**Author:**  [@marshmellow77](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmarshmellow77)%5C
**Post date:** [March 17, 2022, 5:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-4-9-0-on-sagemaker%2F9145%2F5 "2022-03-17T17:30:27Z")
Hi Paras, there are indeed many newer versions of the HF Sagemaker DLCs out, see here for reference: [Reference](https://huggingface.co/docs/sagemaker/reference#inference-dlc-overview)
Cheers
Heiko
---
**Author:**  [@philschmid](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fphilschmid)%5C
**Post date:** [March 17, 2022, 5:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-4-9-0-on-sagemaker%2F9145%2F6 "2022-03-17T17:39:18Z")
Hey @parasmehta,
currently, ASR is not yet supported for zero-code deployments.
We are working on it to ship with the next version of the inference toolkit hopefully within 2-4 weeks. Until that, you can create a custom `inference.py` for it.
You can follow this example [notebooks/sagemaker-notebook.ipynb at master · huggingface/notebooks · GitHub](https://github.com/huggingface/notebooks/blob/master/sagemaker/17_custom_inference_script/sagemaker-notebook.ipynb) since you are using audio data you additionally need to adjust the `input_fn` .
---
**Author:**  [@parasmehta](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fparasmehta)%5C
**Post date:** [March 22, 2022, 4:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-4-9-0-on-sagemaker%2F9145%2F7 "2022-03-22T16:14:41Z")
Thank you for your answers @philschmid and @marshmellow77
I was very happy to see the list of Sagemaker DLCs, but then I realised that the version of transformers that I want to use (`v4.15.0`) is not yet supported in the latest DLCs 😢 They support up to transformers 4.12.3 currently.
So, what are my options if I want to use a model from transformers v4.15.0 for inference on Sagemaker? If you could point me to any tutorials/documentation on this, that would be helpful too. Thanks!
---
**Author:**  [@philschmid](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fphilschmid)%5C
**Post date:** [March 22, 2022, 4:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-4-9-0-on-sagemaker%2F9145%2F8 "2022-03-22T16:30:31Z")
Hey @parasmehta,
we are in the middle of releasing a new DLC with transformers version `4.17.0` you can follow this PR: [feature: Hugging Face Transformers 4.17 for PT 1.10 by saimidu · Pull Request #3011 · aws/sagemaker-python-sdk · GitHub](https://github.com/aws/sagemaker-python-sdk/pull/3011)
In the meantime, you can add a custom `requirements.txt` and provide the version you want to use.
---
**Author:**  [@parasmehta](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fparasmehta)%5C
**Post date:** [March 23, 2022, 6:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-4-9-0-on-sagemaker%2F9145%2F9 "2022-03-23T18:57:46Z")
Thank you for the pointer @philschmid. In that PR, I noticed that some containers for version `4.17.0` have already been released as mentioned here: [Release v1.0-hf-4.17.0-pt-1.10.2-py38 · aws/deep-learning-containers · GitHub](https://github.com/aws/deep-learning-containers/releases/tag/v1.0-hf-4.17.0-pt-1.10.2-py38)
This nearly made my day 😃 I tried to use one of these images and run inference on it for ASR using a simple wav file like this:
```auto
from sagemaker.huggingface import HuggingFaceModel
import sagemaker
role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models
hub = {
'HF_MODEL_ID':'facebook/wav2vec2-xlsr-53-espeak-cv-ft',
'HF_TASK':'automatic-speech-recognition'
}
# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
transformers_version='4.17.0',
pytorch_version='1.10.2',
py_version='py38',
image_uri='763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-inference:1.10.2-transformers4.17.0-cpu-py38-ubuntu20.04-v1.0',
env=hub,
role=role,
)
# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
initial_instance_count=1, # number of instances
instance_type='ml.m5.xlarge' # ec2 instance type
)
result = predictor.predict({"inputs":"audio.wav"})
print(result)
```
But the `predict` function throws an Exception:
```auto
ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{
"code": 400,
"type": "InternalServerException",
"message": "[Errno 2] No such file or directory: \u0027audio.wav\u0027"
```
The `audio.wav` file is in the same folder as the Jupyter notebook. I can’t figure out the reason for this Exception. Any clue what might be going wrong? Thanks!
---
**Author:**  [@philschmid](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fphilschmid)%5C
**Post date:** [March 24, 2022, 8:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-4-9-0-on-sagemaker%2F9145%2F10 "2022-03-24T08:07:47Z")
@parasmehta you are sending as input a string “audio.wav” not any audio data at all. You can find documentation on how the predictor works here: [Predictors — sagemaker 2.80.0 documentation](https://sagemaker.readthedocs.io/en/stable/api/inference/predictors.html)
---
**Author:**  [@parasmehta](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fparasmehta)%5C
**Post date:** [March 24, 2022, 12:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-4-9-0-on-sagemaker%2F9145%2F11 "2022-03-24T12:41:06Z")
@philschmid Are you sure that’s the problem? Because the Exception says that it couldn’t find the file or folder. So, it expects a file. The documentation for `pipelines` also says that `inputs` can be either the filename, bytes or numpy ndarray ([Pipelines](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline). **call** ).
Anyways, I tried supplying the predict function with the audio data as bytes and numpy ndarray. But I still get errors.
When I try with `bytes`:
Code:
```auto
with open("audio.wav", "rb") as f:
data = f.read()
predictor.predict({"inputs": data})
```
Exception:
```auto
TypeError: Object of type 'bytes' is not JSON serializable
```
When I try with a numpy ndarray:
Code:
```auto
import torchaudio
speech_array, _sampling_rate = torchaudio.load("audio.wav")
resampler = torchaudio.transforms.Resample(_sampling_rate, 16000)
speech = resampler(speech_array).squeeze().numpy()
result = predictor.predict({"inputs": speech})
```
Exception:
```auto
An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{
"code": 400,
"type": "InternalServerException",
"message": "We expect a numpy ndarray as input, got `\u003cclass \u0027float\u0027\u003e`"
}
```
I am a bit clueless and running out of ideas now.
---
**Author:**  [@philschmid](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fphilschmid)%5C
**Post date:** [March 25, 2022, 8:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-4-9-0-on-sagemaker%2F9145%2F12 "2022-03-25T08:15:10Z")
I am if you check the documentation of the ASR Pipeline it says
> - **inputs** ( `np.ndarray` or `bytes` or `str` or `dict` ) — The inputs is either :
> - `str` that is the filename of the audio file, the file will be read at the correct sampling rate to get the waveform using _ffmpeg_ . This requires _ffmpeg_ to be installed on the system.
> - `bytes` it is supposed to be the content of an audio file and is interpreted by _ffmpeg_ in the same way.
> - ( `np.ndarray` of shape (n, ) of type `np.float32` or `np.float64` ) Raw audio at the correct sampling rate (no further check will be done)
> - `dict` form can be used to pass raw audio sampled at arbitrary `sampling_rate` and let this pipeline do the resampling. The dict must be in the format `{"sampling_rate": int, "raw": np.array}` with optionally a `"stride": (left: int, right: int)` than can ask the pipeline to treat the first `left` samples and last `right` samples to be ignored in decoding (but used at inference to provide more context to the model). Only use `stride` with CTC models.
You are providing a `str` so SageMaker tries to load the `audio.wav` from disk in the endpoint which of course didn’t exist.
But i am happy to share that we are working on a new DLC version, which makes this much easier
---
**Author:**  [@parasmehta](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fparasmehta)%5C
**Post date:** [March 25, 2022, 8:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-4-9-0-on-sagemaker%2F9145%2F13 "2022-03-25T08:40:05Z")
Ah, I see. Any ideas then why sending bytes or numpy ndarray doesn’t work?