# Gpt2 inference with onnx and quantize
**URL:**
**Author:**  [@yanagar25](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fyanagar25)%5C
**Post date:** [February 1, 2021, 7:42am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-inference-with-onnx-and-quantize%2F3459%2F1 "2021-02-01T07:42:01Z")
Hey guys,
I’ve managed to create a quantize version of gpt2 using onnxruntime but i don’t seem to be able to run it for some reason. anyone has a tutorial for it? also how does the “generate” method of the model will work with that any ideas?
---
**Author:**  [@lewtun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flewtun)%5C
**Post date:** [February 1, 2021, 8:42am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-inference-with-onnx-and-quantize%2F3459%2F2 "2021-02-01T08:42:25Z")
Hi @yanagar25 when you say you cannot run the quantized version, what kind of error are you running into?
Here’s a notebook that explains how to export a pretrained model to the ONNX format: [transformers/04-onnx-export.ipynb at master · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb)
You can also find more details here: [Exporting transformers models — transformers 4.2.0 documentation](https://huggingface.co/transformers/serialization.html?highlight=generate)
I don’t see an obvious reason why the `generate` method should not work after quantization, so as with most things in deep learning the best advice is to just try and see if it does 🙂
---
**Author:**  [@yanagar25](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fyanagar25)%5C
**Post date:** [February 2, 2021, 11:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-inference-with-onnx-and-quantize%2F3459%2F3 "2021-02-02T11:15:41Z")
Hey, @lewtun
I am sorry if i am asking something that have been answered before. but in order to run the quantized model i need to run it with `onnxruntime.InferenceSession`. How does that can be combined with using the `generate` method? from what i understand i need to copy the entire logic from the `generate` method and instead of using `self(...)` use `session.run(None, ort_inputs)`. please correct me if i’m wrong
---
**Author:**  [@lewtun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flewtun)%5C
**Post date:** [February 3, 2021, 10:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-inference-with-onnx-and-quantize%2F3459%2F4 "2021-02-03T10:09:08Z")
Ah now I understand better what you’re trying to achieve. Indeed you might have to write your own `generate` method so that you can integrate the `InferenceSession` - there’s an example of doing text generation with GPT-2 in the ONNX repo here: [onnxruntime/Inference\_GPT2\_with\_OnnxRuntime\_on\_CPU.ipynb at master · microsoft/onnxruntime · GitHub](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/transformers/notebooks/Inference_GPT2_with_OnnxRuntime_on_CPU.ipynb)
You could just adapt their approach to include the generation method you need (beam search, sampling etc)
---
**Author:**  [@yanagar25](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fyanagar25)%5C
**Post date:** [February 3, 2021, 10:43am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-inference-with-onnx-and-quantize%2F3459%2F5 "2021-02-03T10:43:24Z")
thank you so much for the reply! 🙂
---
**Author:**  [@lewtun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flewtun)%5C
**Post date:** [February 3, 2021, 10:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-inference-with-onnx-and-quantize%2F3459%2F6 "2021-02-03T10:58:02Z")
FYI there’s a nice section in the docs that explains the various text generation strategies and how they’re implemented: [Utilities for Generation — transformers 4.2.0 documentation](https://huggingface.co/transformers/internal/generation_utils.html)
---
**Author:**  [@yanagar25](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fyanagar25)%5C
**Post date:** [February 3, 2021, 11:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-inference-with-onnx-and-quantize%2F3459%2F7 "2021-02-03T11:29:23Z")
I will definitely look into it! thank you again 🙂