# T5 inference performance
**URL:**
**Author:**  [@GenV](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGenV)%5C
**Post date:** [March 8, 2022, 9:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-inference-performance%2F15494%2F1 "2022-03-08T09:53:41Z")
Hello,
I would like to improve the inference time of a finetuned T5-base (translation task). I am currently using the `.bin` file (`from_pretrained`) and a GPU. I have tried several approaches, such as ONNX and TensorRT. Having a `max_length=1024`, these approaches perform worse (and sometimes by a lot). Are there any techniques that could help? Thank you.
---
**Author:**  [@YannAgora](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FYannAgora)%5C
**Post date:** [March 8, 2022, 1:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-inference-performance%2F15494%2F2 "2022-03-08T13:31:02Z")
Hey 👋,
Did you try quantization ?
There is an example for pegasus model [here](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpegasus-model-weights-compression-pruning%2F6381%2F9). I tried and it performed pretty well for summarization with an inference time decrease by 2x or 3x
---
**Author:**  [@GenV](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGenV)%5C
**Post date:** [March 8, 2022, 1:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-inference-performance%2F15494%2F3 "2022-03-08T13:53:43Z")
Thanks @YannAgora. Can run with 2x or 3x on GPU?
---
**Author:**  [@YannAgora](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FYannAgora)%5C
**Post date:** [March 8, 2022, 2:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-inference-performance%2F15494%2F4 "2022-03-08T14:05:35Z")
I haven’t tried on a GPU instance but I don’t see why it wouldn’t work.
---
**Author:**  [@GenV](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGenV)%5C
**Post date:** [March 8, 2022, 2:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-inference-performance%2F15494%2F5 "2022-03-08T14:28:19Z")
@YannAgora I got this error using the code, adding `reconstructed_quantized_model.to("cuda")` for GPU inference:
`NotImplementedError: Could not run 'quantized::linear_dynamic' with arguments from the 'CUDA' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'quantized::linear_dynamic' is only available for these backends: [CPU, BackendSelect, Python, Named, Conjugate, Negative, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradLazy, AutogradXPU, AutogradMLC, Tracer, UNKNOWN_TENSOR_TYPE_ID, Autocast, Batched, VmapMode].`
I saw that GPU is not supported [here](https://discuss.pytorch.org/t/does-dynamic-quantization-support-gpu/119231).
---
**Author:**  [@YannAgora](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FYannAgora)%5C
**Post date:** [March 8, 2022, 2:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-inference-performance%2F15494%2F6 "2022-03-08T14:41:43Z")
Oh ok I didn’t know that 😅