# Using onnx for text-generation with GPT-2
**URL:**
**Author:**  [@saied](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsaied)%5C
**Post date:** [November 1, 2021, 12:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-onnx-for-text-generation-with-gpt-2%2F11253%2F1 "2021-11-01T12:37:51Z")
Hi @valhalla @patrickvonplaten , I was working with [onnx\_transformers](https://github.com/patil-suraj/onnx_transformers) and using onnx for GPT-2 model and text-generation task. I used transformer pipeline for text-generation and the runtime for generating text was a bit high(20~30s) and I’ve tried using different approaches like using cronjobs to handle it but it didn’t help. and I found your repo and think of using onnx to accelerate the text generation. As I read the README on the repo there is no text-generation for onnx\_transformers. I also used some mehtods in this notebook: [Inference\_GPT2\_with\_OnnxRuntime\_on\_CPU](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/transformers/notebooks/Inference_GPT2_with_OnnxRuntime_on_CPU.ipynb) but the qulity of generated text was not even near transformer pipline, would you please give me some insight about this runtime issue and how can I accelerate text-generation besides increasing resources.
Thanks 🙂
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [November 1, 2021, 4:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-onnx-for-text-generation-with-gpt-2%2F11253%2F2 "2021-11-01T16:00:51Z")
Hi,
We’ve recently added an example of exporting BART with ONNX, including beam search generation: [https://github.com/huggingface/transformers/tree/master/examples/onnx/pytorch/translation](https://github.com/huggingface/transformers/tree/master/examples/onnx/pytorch/translation)
However, it doesn’t include a README right now, which could be very useful to explain how exactly the model can be used. I’ve asked the author to add it.
---
**Author:**  [@saied](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsaied)%5C
**Post date:** [November 2, 2021, 7:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-onnx-for-text-generation-with-gpt-2%2F11253%2F3 "2021-11-02T07:33:34Z")
Thnaks @nielsr
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [July 25, 2022, 11:35am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-onnx-for-text-generation-with-gpt-2%2F11253%2F4 "2022-07-25T11:35:37Z")
The new URL is here: [transformers/examples/research\_projects/onnx/summarization at main · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/tree/main/examples/research_projects/onnx/summarization)
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [February 3, 2023, 9:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-onnx-for-text-generation-with-gpt-2%2F11253%2F5 "2023-02-03T09:25:08Z")
Update here; text generation with ONNX models is now natively supported in [HuggingFace Optimum](https://huggingface.co/docs/optimum/index). This library is meant for optimization/pruning/quantization of Transformer based models to run on all kinds of hardware.
For ONNX, the library implements several ONNX-counterpart classes of the classes available in Transformers. For instance, BertModel is called `ORTModel` in Optimum (ORT = ONNX Runtime).
Check the guide here: [Overview](https://huggingface.co/docs/optimum/onnxruntime/overview)