# How to successfully ONNX pretrained models
**URL:**
**Author:**  [@MH-BS](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMH-BS)%5C
**Post date:** [January 26, 2025, 11:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-successfully-onnx-pretrained-models%2F137390%2F1 "2025-01-26T11:53:20Z")
Hi!
I was looking for a way to use multithreading ([How to use multithreading on a CPU - #6 by MH-BS](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-multithreading-on-a-cpu%2F137279%2F6)) and was directed towards ONNX. As I understood it converts existing models and then they could use multiple threads on my CPU. Unfortunately I always encounter problems. I know that we cannot solve all of them here, but I thought, I will share two or three of them and maybe I can learn from your advice, what I am doing wrong.
---
**Author:**  [@MH-BS](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMH-BS)%5C
**Post date:** [January 26, 2025, 11:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-successfully-onnx-pretrained-models%2F137390%2F2 "2025-01-26T11:56:12Z")
The first one is about **Llava-llama-3-8b-v1-1**. The script looks like this:
```auto
cat conv.py
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
import onnx
from onnxruntime import InferenceSession
from PIL import Image
def convert_to_onnx(model, tokenizer, model_name='test.onnx'):
dummy_input = tokenizer("Inserting some dummy text here since I don't have your actual input.",
return_tensors='pt').input_ids
torch.onnx.export(
model, # model being run
dummy_input, # model input (or a tuple for multiple inputs)
model_name, # where to save the model (can be a file or file-like object)
export_params=True, # store the trained parameter weights inside the model file
opset_version=11, # the ONNX version to export the model to
do_constant_folding=True # whether to execute constant folding for optimization
)
print(f"Model converted to {model_name}.")
def process_model(onnx_model, image, frage):
print("\nQuestion: " + question)
# Placeholder for actual InferenceSession and processing
session = InferenceSession(onnx_model)
input_name = session.get_inputs()[0].name
#image_flags = torch.Tensor([0, 1, 2]) # Example initialization
#image_flags = image_flags.squeeze(-1)
# Placeholder: Convert your image into the required input format
# result = session.run(None, {input_name: image}) [adjust this according to your input type]
# print(result)
path="xtuner/llava-llama-3-8b-v1_1"
model = AutoModelForCausalLM.from_pretrained(path, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
convert_to_onnx(model, tokenizer)
image = Image.open("tmp2.jpg")
image.show()
question = "Please describe the image."
process_model("test.onnx", image, frage)
```
The output looks like this:
[…]
odel-00009-of-00009.safetensors: 92%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 965M/1.05G [02:16\<00:12, 7.1model-00009-of-00009.safetensors: 93%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 975M/1.05G [02:17\<00:10, 7.1model-00009-of-00009.safetensors: 94%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 986M/1.05G [02:19\<00:09, 7.1model-00009-of-00009.safetensors: 95%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 996M/1.05G [02:20\<00:07, 6.8model-00009-of-00009.safetensors: 96%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 1.01G/1.05G [02:22\<00:06, 6.9model-00009-of-00009.safetensors: 97%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 1.02G/1.05G [02:23\<00:04, 7.0model-00009-of-00009.safetensors: 98%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 1.03G/1.05G [02:25\<00:03, 7.0model-00009-of-00009.safetensors: 99%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 1.04G/1.05G [02:26\<00:01, 7.1model-00009-of-00009.safetensors: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊| 1.05G/1.05G [02:28\<00:00, 7.1model-00009-of-00009.safetensors: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.05G/1.05G [02:28\<00:00, 7.08MB/s]
Downloading shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [37:50\<00:00, 252.25s/it]
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:31\<00:00, 3.50s/it]
generation\_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 126/126 [00:00\<00:00, 715kB/s]
tokenizer\_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 51.0k/51.0k [00:00\<00:00, 909kB/s]
tokenizer.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9.08M/9.08M [00:01\<00:00, 6.14MB/s]
special\_tokens\_map.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 301/301 [00:00\<00:00, 1.78MB/s]
/home/martin/esn\_vqa/lib/python3.12/site-packages/transformers/models/llama/modeling\_llama.py:726: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if sequence\_length != 1:
Traceback (most recent call last):
File “/home/martin/esn\_vqa/conv\_moon.py”, line 35, in
convert\_to\_onnx(model, tokenizer)
File “/home/martin/esn\_vqa/conv\_moon.py”, line 10, in convert\_to\_onnx
torch.onnx.export(
File “/home/martin/esn\_vqa/lib/python3.12/site-packages/torch/onnx/ **init**.py”, line 375, in export
export(
File “/home/martin/esn\_vqa/lib/python3.12/site-packages/torch/onnx/utils.py”, line 502, in export
\_export(
File “/home/martin/esn\_vqa/lib/python3.12/site-packages/torch/onnx/utils.py”, line 1564, in \_export
graph, params\_dict, torch\_out = \_model\_to\_graph(
^^^^^^^^^^^^^^^^
File “/home/martin/esn\_vqa/lib/python3.12/site-packages/torch/onnx/utils.py”, line 1113, in \_model\_to\_graph
graph, params, torch\_out, module = \_create\_jit\_graph(model, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/martin/esn\_vqa/lib/python3.12/site-packages/torch/onnx/utils.py”, line 997, in \_create\_jit\_graph
graph, torch\_out = \_trace\_and\_get\_graph\_from\_model(model, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/martin/esn\_vqa/lib/python3.12/site-packages/torch/onnx/utils.py”, line 904, in \_trace\_and\_get\_graph\_from\_model
trace\_graph, torch\_out, inputs\_states = torch.jit.\_get\_trace\_graph(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/martin/esn\_vqa/lib/python3.12/site-packages/torch/jit/\_trace.py”, line 1500, in \_get\_trace\_graph
outs = ONNXTracedModule(
^^^^^^^^^^^^^^^^^
File “/home/martin/esn\_vqa/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1736, in \_wrapped\_call\_impl
return self.\_call\_impl(\*args, \*\*kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/martin/esn\_vqa/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1747, in \_call\_impl
return forward\_call(\*args, \*\*kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/martin/esn\_vqa/lib/python3.12/site-packages/torch/jit/\_trace.py”, line 139, in forward
graph, out = torch.\_C.\_create\_graph\_by\_tracing(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/martin/esn\_vqa/lib/python3.12/site-packages/torch/jit/\_trace.py”, line 133, in wrapper
out\_vars, \_ = \_flatten(outs)
^^^^^^^^^^^^^^
RuntimeError: Only tuples, lists and Variables are supported as JIT inputs/outputs. Dictionaries and strings are also accepted, but their usage is not recommended. Here, received an input of unsupported type: DynamicCache
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [January 26, 2025, 11:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-successfully-onnx-pretrained-models%2F137390%2F3 "2025-01-26T11:56:45Z")
If you have any questions about ONNX, it’s a good idea to ask them on the ONNX Community. I might be able to help you with some basic mistakes…
> **[onnx-community (ONNX Community)](https://huggingface.co/onnx-community)**
>
> Org profile for ONNX Community on Hugging Face, the AI community building the future.
---
**Author:**  [@MH-BS](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMH-BS)%5C
**Post date:** [January 26, 2025, 11:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-successfully-onnx-pretrained-models%2F137390%2F4 "2025-01-26T11:58:37Z")
Ah okay, do you think, I should not proceed here? I already wanted to post the next model experience. But of course I can also switch to the ONNX community.
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [January 26, 2025, 12:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-successfully-onnx-pretrained-models%2F137390%2F5 "2025-01-26T12:01:02Z")
No, it’s genuinely helpful that you’re sharing your experiences with everyone.😅
I’ve just explained it. By the way, from what I’ve seen so far, I think there are a lot of errors that occur when the ONNX library is old, but is this happening with the latest version of ONNX?
Edit:
Also, I think this is probably the recommended method for exporting now.
> **[Export a model to ONNX with optimum.exporters.onnx](https://huggingface.co/docs/optimum/exporters/onnx/usage_guides/export_a_model)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
---
**Author:**  [@MH-BS](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMH-BS)%5C
**Post date:** [January 26, 2025, 12:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-successfully-onnx-pretrained-models%2F137390%2F6 "2025-01-26T12:05:13Z")
Thank you, I will take a look at the optimum approach.
---
**Author:**  [@MH-BS](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMH-BS)%5C
**Post date:** [January 26, 2025, 12:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-successfully-onnx-pretrained-models%2F137390%2F7 "2025-01-26T12:22:14Z")
It may be that the optimum approach is not suitable. I just got this response:
> ValueError: Asked to export a llama model for the task image-text-to-text (auto-detected), but the Optimum ONNX exporter only supports the tasks feature-extraction, feature-extraction-with-past, text-generation, text-generation-with-past, text-classification for llama. Please use a supported task. Please open an issue at [GitHub · Where software is built](https://github.com/huggingface/optimum/issues) if you would like the task image-text-to-text to be supported in the ONNX export for llama.
And as all the models I am look at right now are from the VQA/VL area, I guess that optimum is not suited for that task.
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [January 26, 2025, 1:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-successfully-onnx-pretrained-models%2F137390%2F8 "2025-01-26T13:03:59Z")
It seems that in some cases, this can be avoided by directly specifying the task. If that doesn’t work, let’s look for something other than ONNX and Optimum.
>
>
> \### Feature request
>
> I wonder if the task text-classification can to be supporte…d in the ONNX export for clip? Ich want to use the openai/clip-vit-large-path14 model for zero-shot image classification (classification of images without pretraining based on given candidate labels) but I get the following error:
> ValueError Traceback (most recent call last)
> File /home/danne00a/ZablageBlazeG/ZeroShotClassification/zeroshotclassifier.py:2
> \[1\](https://vscode-remote+ssh-002dremote-002bdemlhr7sw119x.vscode-resource.vscode-cdn.net/home/danne00a/ZablageBlazeG/ZeroShotClassification/zeroshotclassifier.py:1) #%%
> \----\> \[2\](https://vscode-remote+ssh-002dremote-002bdemlhr7sw119x.vscode-resource.vscode-cdn.net/home/danne00a/ZablageBlazeG/ZeroShotClassification/zeroshotclassifier.py:2) ort\_model = ORTModelForSequenceClassification.from\_pretrained(model\_checkpoint, export=True)
>
> File ~/mambaforge/envs/ZeroShot\_Mamba\_env/lib/python3.11/site-packages/optimum/onnxruntime/modeling\_ort.py:669, in ORTModel.from\_pretrained(cls, model\_id, export, force\_download, use\_auth\_token, cache\_dir, subfolder, config, local\_files\_only, provider, session\_options, provider\_options, use\_io\_binding, \*\*kwargs)
> \[620\](https://vscode-remote+ssh-002dremote-002bdemlhr7sw119x.vscode-resource.vscode-cdn.net/home/danne00a/ZablageBlazeG/ZeroShotClassification/~/mambaforge/envs/ZeroShot\_Mamba\_env/lib/python3.11/site-packages/optimum/onnxruntime/modeling\_ort.py:620) @classmethod
> \[621\](https://vscode-remote+ssh-002dremote-002bdemlhr7sw119x.vscode-resource.vscode-cdn.net/home/danne00a/ZablageBlazeG/ZeroShotClassification/~/mambaforge/envs/ZeroShot\_Mamba\_env/lib/python3.11/site-packages/optimum/onnxruntime/modeling\_ort.py:621) @add\_start\_docstrings(FROM\_PRETRAINED\_START\_DOCSTRING)
> \[622\](https://vscode-remote+ssh-002dremote-002bdemlhr7sw119x.vscode-resource.vscode-cdn.net/home/danne00a/ZablageBlazeG/ZeroShotClassification/~/mambaforge/envs/ZeroShot\_Mamba\_env/lib/python3.11/site-packages/optimum/onnxruntime/modeling\_ort.py:622) def from\_pretrained(
> (...)
> \[636\](https://vscode-remote+ssh-002dremote-002bdemlhr7sw119x.vscode-resource.vscode-cdn.net/home/danne00a/ZablageBlazeG/ZeroShotClassification/~/mambaforge/envs/ZeroShot\_Mamba\_env/lib/python3.11/site-packages/optimum/onnxruntime/modeling\_ort.py:636) \*\*kwargs,
> \[637\](https://vscode-remote+ssh-002dremote-002bdemlhr7sw119x.vscode-resource.vscode-cdn.net/home/danne00a/ZablageBlazeG/ZeroShotClassification/~/mambaforge/envs/ZeroShot\_Mamba\_env/lib/python3.11/site-packages/optimum/onnxruntime/modeling\_ort.py:637) ):
> \[638\](https://vscode-remote+ssh-002dremote-002bdemlhr7sw119x.vscode-resource.vscode-cdn.net/home/danne00a/ZablageBlazeG/ZeroShotClassification/~/mambaforge/envs/ZeroShot\_Mamba\_env/lib/python3.11/site-packages/optimum/onnxruntime/modeling\_ort.py:638) """
> \[639\](https://vscode-remote+ssh-002dremote-002bdemlhr7sw119x.vscode-resource.vscode-cdn.net/home/danne00a/ZablageBlazeG/ZeroShotClassification/~/mambaforge/envs/ZeroShot\_Mamba\_env/lib/python3.11/site-packages/optimum/onnxruntime/modeling\_ort.py:639) provider (\`str\`, defaults to \`"CPUExecutionProvider"\`):
> \[640\](https://vscode-remote+ssh-002dremote-002bdemlhr7sw119x.vscode-resource.vscode-cdn.net/home/danne00a/ZablageBlazeG/ZeroShotClassification/~/mambaforge/envs/ZeroShot\_Mamba\_env/lib/python3.11/site-packages/optimum/onnxruntime/modeling\_ort.py:640) ONNX Runtime provider to use for loading the model. See https://onnxruntime.ai/docs/execution-providers/ for
> (...)
> \[667\](https://vscode-remote+ssh-002dremote-002bdemlhr7sw119x.vscode-resource.vscode-cdn.net/home/danne00a/ZablageBlazeG/ZeroShotClassification/~/mambaforge/envs/ZeroShot\_Mamba\_env/lib/python3.11/site-packages/optimum/onnxruntime/modeling\_ort.py:667) \`ORTModel\`: The loaded ORTModel model.
> \[668\](https://vscode-remote+ssh-002dremote-002bdemlhr7sw119x.vscode-resource.vscode-cdn.net/home/danne00a/ZablageBlazeG/ZeroShotClassification/~/mambaforge/envs/ZeroShot\_Mamba\_env/lib/python3.11/site-packages/optimum/onnxruntime/modeling\_ort.py:668) """
> \--\> \[669\](https://vscode-remote+ssh-002dremote-002bdemlhr7sw119x.vscode-resource.vscode-cdn.net/home/danne00a/ZablageBlazeG/ZeroShotClassification/~/mambaforge/envs/ZeroShot\_Mamba\_env/lib/python3.11/site-packages/optimum/onnxruntime/modeling\_ort.py:669) return super().from\_pretrained(
> \[670\](https://vscode-remote+ssh-002dremote-002bdemlhr7sw119x.vscode-resource.vscode-cdn.net/home/danne00a/ZablageBlazeG/ZeroShotClassification/~/mambaforge/envs/ZeroShot\_Mamba\_env/lib/python3.11/site-packages/optimum/onnxruntime/modeling\_ort.py:670) model\_id,
> \[671\](https://vscode-remote+ssh-002dremote-002bdemlhr7sw119x.vscode-resource.vscode-cdn.net/home/danne00a/ZablageBlazeG/ZeroShotClassification/~/mambaforge/envs/ZeroShot\_Mamba\_env/lib/python3.11/site-packages/optimum/onnxruntime/modeling\_ort.py:671) export=export,
> \[672\](https://vscode-remote+ssh-002dremote-002bdemlhr7sw119x.vscode-resource.vscode-cdn.net/home/danne00a/ZablageBlazeG/ZeroShotClassification/~/mambaforge/envs/ZeroShot\_Mamba\_env/lib/python3.11/site-packages/optimum/onnxruntime/modeling\_ort.py:672) force\_download=force\_download,
> \[673\](https://vscode-remote+ssh-002dremote-002bdemlhr7sw119x.vscode-resource.vscode-cdn.net/home/danne00a/ZablageBlazeG/ZeroShotClassification/~/mambaforge/envs/ZeroShot\_Mamba\_env/lib/python3.11/site-packages/optimum/onnxruntime/modeling\_ort.py:673) use\_auth\_token=use\_auth\_token,
> \[674\](https://vscode-remote+ssh-002dremote-002bdemlhr7sw119x.vscode-resource.vscode-cdn.net/home/danne00a/ZablageBlazeG/ZeroShotClassification/~/mambaforge/envs/ZeroShot\_Mamba\_env/lib/python3.11/site-packages/optimum/onnxruntime/modeling\_ort.py:674) cache\_dir=cache\_dir,
> ...
> \[274\](https://vscode-remote+ssh-002dremote-002bdemlhr7sw119x.vscode-resource.vscode-cdn.net/home/danne00a/ZablageBlazeG/ZeroShotClassification/~/mambaforge/envs/ZeroShot\_Mamba\_env/lib/python3.11/site-packages/optimum/exporters/onnx/\_\_main\_\_.py:274) )
> \[276\](https://vscode-remote+ssh-002dremote-002bdemlhr7sw119x.vscode-resource.vscode-cdn.net/home/danne00a/ZablageBlazeG/ZeroShotClassification/~/mambaforge/envs/ZeroShot\_Mamba\_env/lib/python3.11/site-packages/optimum/exporters/onnx/\_\_main\_\_.py:276) # TODO: Fix in Transformers so that SdpaAttention class can be exported to ONNX. \`attn\_implementation\` is introduced in Transformers 4.36.
> \[277\](https://vscode-remote+ssh-002dremote-002bdemlhr7sw119x.vscode-resource.vscode-cdn.net/home/danne00a/ZablageBlazeG/ZeroShotClassification/~/mambaforge/envs/ZeroShot\_Mamba\_env/lib/python3.11/site-packages/optimum/exporters/onnx/\_\_main\_\_.py:277) if model\_type in SDPA\_ARCHS\_ONNX\_EXPORT\_NOT\_SUPPORTED and \_transformers\_version \>= version.parse("4.35.99"):
>
> ValueError: Asked to export a clip model for the task text-classification, but the Optimum ONNX exporter only supports the tasks feature-extraction, zero-shot-image-classification for clip. Please use a supported task. Please open an issue at https://github.com/huggingface/optimum/issues if you would like the task text-classification to be supported in the ONNX export for clip.
>
> \### Motivation
>
> I'm struggling with the sioze of the openai/clip-vit-large-patch14 model, thus I want to convert it to OPTIMUM onnx!
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> \### Your contribution
>
> no ideas so far..