# ORT CLI vs. Programmatic

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fort-cli-vs-programmatic%2F46719%3E%5C
**Category:** 🤗Optimum\
**Created:** [July 13, 2023, 4:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fort-cli-vs-programmatic%2F46719 "2023-07-13T16:45:47Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Borell](https://avatars.discourse-cdn.com/v4/letter/b/59ef9b/32.png) [@Borell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBorell)%5C
**Post date:** [July 13, 2023, 4:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fort-cli-vs-programmatic%2F46719%2F1 "2023-07-13T16:45:47Z")

</div>

Hi,

As an experiment, I am running a conversion/optimization from PyTorch to ONNX with dolly-v2-3b. The CLI approach ran successfully for optimization O3 on a Dell 7540 (Xeon E-2286M x 16; 64GB memory). It resulted in a 15% reduction when compared to a not optimized conversion.

When attempting the same via a programmatic approach the program gets killed by the Kernel as it exceeds available memory resources (100% RAM and Swap). This happens when the optimze() method of ORTOptimizer is called. The last output seen is “Optimizing model…”. The configuration was generated with AutoOptimizationConfigO3().

With the CLI execution, memory use never exceeds 80%, with about 25% of Swap. Hence, I have a few questions:

Is there a guide as to required resources for programmatic and CLI ORT Optimum execution?  
What makes it require more resources under the API based optimization?  
In both cases optimization is the same with Level O3, is there more optimization techniques applied with the programmatic approach as compared to the CLI approach?

Thanks,  
Borell

---

<div class="post-metadata">

**Author:** ![regisss](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/regisss/32/22467_2.png) [@regisss](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fregisss)%5C
**Post date:** [July 13, 2023, 6:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fort-cli-vs-programmatic%2F46719%2F2 "2023-07-13T18:54:48Z")

</div>

Hi Borell! Could you share the CLI commands and the script you used to compare both approaches please?

---

<div class="post-metadata">

**Author:** ![Borell](https://avatars.discourse-cdn.com/v4/letter/b/59ef9b/32.png) [@Borell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBorell)%5C
**Post date:** [July 13, 2023, 8:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fort-cli-vs-programmatic%2F46719%2F3 "2023-07-13T20:48:25Z")

</div>

Hi regisss,

Thanks for the reply, I can certainly send you the code. I wrote a Python module which runs CLI or optionally programmatic code. Doesn’t seem that there is an option to attach files. I can paste the code in a reply if that is OK, it is not very long.

Sorry my first time using this forum.

BBorell

---

<div class="post-metadata">

**Author:** ![regisss](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/regisss/32/22467_2.png) [@regisss](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fregisss)%5C
**Post date:** [July 14, 2023, 7:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fort-cli-vs-programmatic%2F46719%2F4 "2023-07-14T07:30:14Z")

</div>

No worries, it’s even better to directly share the code here surrounding your code snippets with ``` 🙂

---

<div class="post-metadata">

**Author:** ![Borell](https://avatars.discourse-cdn.com/v4/letter/b/59ef9b/32.png) [@Borell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBorell)%5C
**Post date:** [July 14, 2023, 1:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fort-cli-vs-programmatic%2F46719%2F5 "2023-07-14T13:17:39Z")

</div>

regisss,

Here is the code. To run with CLI, change cliBool = False to True and change the location path “model\_save”. Comparison of size is then done manually by verifying the size of files created at path model\_save.

The ultimate objective is not Dolly but something like a Flan-T5 XXX. I am exploring the feasibility of using ORT. I attempted earlier with OpenVINO, but their implementation crashed. It is now being evaluated by Intel developers after they confirmed the issue with their own sample code.

‘’’  
#!/usr/bin/env python

#standard imports  
import os  
import subprocess  
from pathlib import Path

#third-party imports  
from transformers import AutoTokenizer  
from optimum.onnxruntime import ORTModelForCausalLM  
from optimum.pipelines import pipeline  
import torch  
from tensorflow.python.ops.initializers\_ns import variables  
#from optimum.onnxruntime import OptimizationConfig, ORTOptimizer #deferred to optimization

cliBool = False

#getting from [huggingface.co](http://huggingface.co)  
model\_remote = ‘databricks/dolly-v2-3b’  
model\_path = Path(‘~/.cache/huggingface/hub/models–databricks–dolly-v2-3b’).expanduser()  
if model\_path.exists():  
print(“\*\*\*\*\*\*\*\*\*\*\*\*\* 1: Getting local PyTorch Model”)  
#… from local cache; process does not expand user path; also, needs to point at config.json file location  
model\_path = model\_path / ‘snapshots/f6c9be08f16fe4d3a719bee0a4a7c7415b5c65df’  
#saving to: /home/jellybean/.cache/huggingface/hub/models–databricks–dolly-v2-3b/snapshots/f6c9be08f16fe4d3a719bee0a4a7c7415b5c65df/optimum/onnx\_rt  
model\_save = model\_path / ‘optimum/onnx\_rt\_optimized’  
else:  
print(“\*\*\*\*\*\*\*\*\*\*\*\*\* 1: Getting Hugging Face PyTorch Model”)  
model\_path = model\_remote  
#mannually move it to cache snapshot upon created  
model\_save = Path(‘~/.cache/huggingface/hub/’).expanduser() / ‘optimum/onnx\_rt’

### GPU not enough memory: the following reports before onnx conversion starts

#mem = torch.cuda.mem\_get\_info()  
#print(f"CUDA Memory - Available: {mem[0]} Total: {mem[1]}“)  
#print(f"CUDA Memory - Total: {torch.cuda.get\_device\_properties(0).total\_memory} Reserved: {torch.cuda.memory\_reserved(0)} Allocated: {torch.cuda.memory\_allocated(0)}”)  
#print(f"{torch.cuda.memory\_summary(device=0, abbreviated=True)}")

# 

#check file exist so we don’t convert again  
config = model\_save / ‘config.json’

‘’‘torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 76.00 MiB…  
… (GPU 0; 5.79 GiB total capacity; 5.26 GiB already allocated; 69.50 MiB free;…  
… 5.28 GiB reserved in total by PyTorch) If reserved memory is \>\> allocated memory…  
… try setting max\_split\_size\_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH\_CUDA\_ALLOC\_CONF…  
… but os.environ[‘PYTORCH\_CUDA\_ALLOC\_CONF’] = ‘max\_split\_size\_mb:21’ still gets OutOfMemoryError  
cli = ‘optimum-cli export onnx -m {model\_path} --cache\_dir {model\_save} --task text-generation  
cli += f’–framework pt --device cuda --optimize O4 --batch\_size 32 {model\_save}’  
Cause: --device cuda --optimize O4  
‘’’  
if not config.exists():  
if cliBool:  
print(“\*\*\*\*\*\*\*\*\*\*\*\*\* 2: CLI-ONNX Convert/Optimize”)  
#switching to CPU; need to specify --task if not from huggingface; optimizing to Level O3…  
#… export dolly to onnx: \> optimum-cli export onnx --help  
cli = f’optimum-cli export onnx -m {model\_path} --task text-generation-with-past --framework pt --no-post-process --optimize O3 {model\_save}’  
with subprocess.Popen([cli], shell=True, stdout=subprocess.PIPE, stdin=subprocess.PIPE) as proc:  
\_, \_ = proc.communicate()  
else:  
print(“\*\*\*\*\*\*\*\*\*\*\*\*\* 2: Programmatic-ONNX Converte/Optimize ONNX”)  
from optimum.onnxruntime import AutoOptimizationConfig, ORTOptimizer

```
	print(" ************* 2.a")
	model = ORTModelForCausalLM.from_pretrained(model_path, export=True)
	print(" ************* 2.b")
	optimizer = ORTOptimizer.from_pretrained(model)
	print(" ************* 2.c")
	#optimizing to Level O3
	optimization_config = AutoOptimizationConfig.O3()
	print(optimization_config)
	raise SystemExit
	print(" ************* 2.d")
	optimizer.optimize(save_dir=model_save, optimization_config=optimization_config)

```

else: print(“\*\*\*\*\*\*\*\*\*\*\*\*\* 2: Skipping ONNX Convertion/Optimization”)

#inference using pt to onnx model  
print(“\*\*\*\*\*\*\*\*\*\*\*\*\* 3: ONNX Model Inference”)  
tokenizer = AutoTokenizer.from\_pretrained(model\_save)  
model\_ort = ORTModelForCausalLM.from\_pretrained(model\_save)  
ort\_pipe = pipeline(‘text-generation’, model=model\_ort, tokenizer=tokenizer, accelerator=‘ort’, framework=‘pt’, device=-1, model\_kwargs={“load\_in\_8bit”: True})  
#try 4 different prompts  
prompt1 = [‘Explain to me what is love.’, ‘You are an idiot’] #produces a list of two lists with dicts  
prompt2 = [‘Explain to me what is hate.’] #produces a list of one list with dict  
prompt3 = ‘Do you love yourself?.’ #produces a list with a dict  
prompt4 = ‘I hate you!’ #produces a list with a dict  
prompts = [prompt1, prompt2, prompt3, prompt4]  
for prompt in prompts:  
res = ort\_pipe(prompt)  
print(f"Prompt: {prompt}“)  
for item in res:  
next\_item = item  
while isinstance(next\_item, list): next\_item = next\_item[0] #loop until no list (looking for dict)  
print(f”\n{next\_item[‘generated\_text’]}")  
else: pass   
else: pass  
‘’’

---

<div class="post-metadata">

**Author:** ![Borell](https://avatars.discourse-cdn.com/v4/letter/b/59ef9b/32.png) [@Borell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBorell)%5C
**Post date:** [July 14, 2023, 1:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fort-cli-vs-programmatic%2F46719%2F6 "2023-07-14T13:38:07Z")

</div>

Here is the code again without getting mangled up.

‘’’  
#!/usr/bin/env python  
#standard imports  
import os  
import subprocess  
from pathlib import Path

devs = subprocess.Popen(‘lspci -nnk | grep -iA2 vga’, shell=True, stdout=subprocess.PIPE).stdout.read().decode(‘utf-8’).split(‘\n’)  
print()  
for dev in devs: print(‘Device:’, dev)  
print(“Info on devices not being used.\n”)

#third-party imports  
from transformers import AutoTokenizer  
from optimum.onnxruntime import ORTModelForCausalLM  
from optimum.pipelines import pipeline  
import torch  
from tensorflow.python.ops.initializers\_ns import variables  
#from optimum.onnxruntime import OptimizationConfig, ORTOptimizer #deferred to optimization

cliBool = False

#getting from [huggingface.co](http://huggingface.co)  
model\_remote = ‘databricks/dolly-v2-3b’  
model\_path = Path(‘~/.cache/huggingface/hub/models–databricks–dolly-v2-3b’).expanduser()  
if model\_path.exists():  
print(“\*\*\*\*\*\*\*\*\*\*\*\*\* 1: Gettting local PyTorch Model”)  
#… from local cache; process does not expand user path; also, needs to point at config.json file location  
model\_path = model\_path / ‘snapshots/f6c9be08f16fe4d3a719bee0a4a7c7415b5c65df’  
#saving to: /home/jellybean/.cache/huggingface/hub/models–databricks–dolly-v2-3b/snapshots/f6c9be08f16fe4d3a719bee0a4a7c7415b5c65df/optimum/onnx\_rt  
model\_save = model\_path / ‘optimum/onnx\_rt\_optimized’  
else:  
print(“\*\*\*\*\*\*\*\*\*\*\*\*\* 1: Getting Hugging Face PyTorch Model”)  
model\_path = model\_remote  
#mannually move it to cache snapshot upon created  
model\_save = Path(‘~/.cache/huggingface/hub/’).expanduser() / ‘optimum/onnx\_rt’

#check file exist so we don’t convert again  
config = model\_save / ‘config.json’

if not config.exists():  
if cliBool:  
print(“\*\*\*\*\*\*\*\*\*\*\*\*\* 2: CLI-ONNX Convert/Optimize”)  
#switching to CPU; need to specify --task if not from huggingface; optimizing to Level O3…  
#… export dolly to onnx: \> optimum-cli export onnx --help  
cli = f’optimum-cli export onnx -m {model\_path} --task text-generation-with-past --framework pt --no-post-process --optimize O3 {model\_save}’  
with subprocess.Popen([cli], shell=True, stdout=subprocess.PIPE, stdin=subprocess.PIPE) as proc:  
\_, \_ = proc.communicate()  
else:  
print(“\*\*\*\*\*\*\*\*\*\*\*\*\* 2: Programmatic-ONNX Convert/Optimize ONNX”)  
from optimum.onnxruntime import AutoOptimizationConfig, ORTOptimizer

```
	print(" ************* 2.a")
	model = ORTModelForCausalLM.from_pretrained(model_path, export=True)
	print(" ************* 2.b")
	optimizer = ORTOptimizer.from_pretrained(model)
	print(" ************* 2.c")
	#optimizing to Level O3
	optimization_config = AutoOptimizationConfig.O3()
	print(optimization_config)
	print(" ************* 2.d")
	optimizer.optimize(save_dir=model_save, optimization_config=optimization_config)

```

else: print(“\*\*\*\*\*\*\*\*\*\*\*\*\* 2: Skipping ONNX Convertion/Optimization”)

#inference using pt to onnx model  
print(“\*\*\*\*\*\*\*\*\*\*\*\*\* 3: ONNX Model Inference”)  
tokenizer = AutoTokenizer.from\_pretrained(model\_save)  
model\_ort = ORTModelForCausalLM.from\_pretrained(model\_save)  
ort\_pipe = pipeline(‘text-generation’, model=model\_ort, tokenizer=tokenizer, accelerator=‘ort’, framework=‘pt’, device=-1, model\_kwargs={“load\_in\_8bit”: True})  
#try 3 different prompts  
prompt1 = [‘Explain to me what is love.’, ‘You are an idiot’] #produces a list of two lists with dicts  
prompt2 = [‘Explain to me what is hate.’] #produces a list of one list with dict  
prompt3 = ‘Do you love yourself?.’ #produces a list with a dict  
prompt4 = ‘I hate you!’ #produces a list with a dict  
prompts = [prompt1, prompt2, prompt3, prompt4]  
for prompt in prompts:  
res = ort\_pipe(prompt)  
print(f"Prompt: {prompt}“)  
for item in res:  
next\_item = item  
while isinstance(next\_item, list): next\_item = next\_item[0] #loop until no list (looking for dict)  
print(f”\n{next\_item[‘generated\_text’]}")  
else: pass   
else: pass  
‘’’

---

<div class="post-metadata">

**Author:** ![fxmarty](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fxmarty/32/23782_2.png) [@fxmarty](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffxmarty)%5C
**Post date:** [July 21, 2023, 11:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fort-cli-vs-programmatic%2F46719%2F7 "2023-07-21T11:46:50Z")

</div>

Hi, a difference between the export CLI and `ORTModel.from_pretrained(..., export=True)` is that by default the CLI fuses the decoder without and with KV cache.

At the time, this feature was quite experimental and we did not make it default in the `from_pretrained` export.

You can try to pass `use_merged=True` to `from_pretrained` to lower the memory usage.

For reference:

> **[Export functions](https://huggingface.co/docs/optimum/main/en/exporters/onnx/package_reference/export#optimum.exporters.onnx.main_export)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

> **[Models](https://huggingface.co/docs/optimum/main/en/onnxruntime/package_reference/modeling_ort#optimum.onnxruntime.ORTModel.from_pretrained.use_merged)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

> **[Release v1.8: extended BetterTransformer support, ONNX merged seq2seq models...](https://github.com/huggingface/optimum/releases/tag/v1.8.2)**
>
> Extended BetterTransformer support
> Various improvements in the PyTorch BetterTransformer integration.
> 
> \[BT\] add BetterTransformer support for ProphetNet by @hirotasoshu in #923
> Improve bettertransf...

See as well the note about the pytorch version here: [Release v1.9: extended ONNX, ONNX Runtime support · huggingface/optimum · GitHub](https://github.com/huggingface/optimum/releases/tag/v1.9.0)

---

<div class="post-metadata">

**Author:** ![Borell](https://avatars.discourse-cdn.com/v4/letter/b/59ef9b/32.png) [@Borell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBorell)%5C
**Post date:** [July 21, 2023, 5:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fort-cli-vs-programmatic%2F46719%2F8 "2023-07-21T17:58:24Z")

</div>

Hi fxmarty,

Thanks for the advice. I added `use\_merged=True. During the execution of from\_pretrained(…) the following error shows:  
Traceback (most recent call last):  
File “/home/jellybean/workspace/careless\_navigator/src/optimum/./dolly\_ort\_optimum.py”, line 87, in   
model = ORTModelForCausalLM.from\_pretrained(model\_path, export=True, use\_merged=True)  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/home/jellybean/.virtualenvs/careless\_navigator/lib/python3.11/site-packages/optimum/onnxruntime/modeling\_ort.py”, line 646, in from\_pretrained  
return super().from\_pretrained(  
^^^^^^^^^^^^^^^^^^^^^^^^  
…  
File “/home/jellybean/.virtualenvs/careless\_navigator/lib64/python3.11/site-packages/torch/onnx/\_internal/onnx\_proto\_utils.py”, line 175, in \_export\_file  
opened\_file.write(model\_bytes)  
OSError: [Errno 28] No space left on device

BTW, this happens with use\_merged=False also. Something has changed since last time I ran the same code. Before I was able to get to the optimizer.optimize(…) step without this error showing. I have moved the model location to a 3TB disk with 1.8TB free space, with same results.

I verified that the CLI execution still runs correctly and produces an optimized ORT model.

Regardless, when I look at the source code for class ORTOptimizer, line 101  
“if model\_or\_path.use\_merged is True:” would raise raise NotImplementedError.

Best,  
Borell

---

<div class="post-metadata">

**Author:** ![Borell](https://avatars.discourse-cdn.com/v4/letter/b/59ef9b/32.png) [@Borell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBorell)%5C
**Post date:** [August 14, 2023, 5:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fort-cli-vs-programmatic%2F46719%2F9 "2023-08-14T17:08:26Z")

</div>

Hi,

I am wondering if this issue is being looked at by anyone, or should it be closed w/o resolution? I will presume that only the CLI version is a workable solution.

Best,  
Borell

---

<div class="post-metadata">

**Author:** ![fxmarty](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fxmarty/32/23782_2.png) [@fxmarty](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffxmarty)%5C
**Post date:** [August 17, 2023, 9:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fort-cli-vs-programmatic%2F46719%2F10 "2023-08-17T09:27:12Z")

</div>

Hi Borell, it should not be the case. Can you open an issue on Github with a reproduction?

---

<div class="post-metadata">

**Author:** ![Borell](https://avatars.discourse-cdn.com/v4/letter/b/59ef9b/32.png) [@Borell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBorell)%5C
**Post date:** [August 17, 2023, 2:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fort-cli-vs-programmatic%2F46719%2F11 "2023-08-17T14:27:47Z")

</div>

@fxmarty ,  
Happy to do that. Can you please clarify on which Github to post?

---

<div class="post-metadata">

**Author:** ![regisss](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/regisss/32/22467_2.png) [@regisss](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fregisss)%5C
**Post date:** [August 17, 2023, 2:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fort-cli-vs-programmatic%2F46719%2F12 "2023-08-17T14:49:36Z")

</div>

@Borell Cool! You can open an issue in the GitHub repo of Optimum: [Issues · huggingface/optimum · GitHub](https://github.com/huggingface/optimum/issues)

---

<div class="post-metadata">

**Author:** ![Borell](https://avatars.discourse-cdn.com/v4/letter/b/59ef9b/32.png) [@Borell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBorell)%5C
**Post date:** [August 17, 2023, 7:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fort-cli-vs-programmatic%2F46719%2F13 "2023-08-17T19:16:58Z")

</div>

@fxmarty, @regisss,

While trying to replicate the issue for posting into GitHub I made a change in the code which seems to have resolved the problem. In the following line,

model = ORTModelForCausalLM.from\_pretrained(model\_path, export=True, use\_merged=True)

I changed use\_merged = False. It was originally set to True as a suggestion to save memory utilization. I don’t fully understand the reason, but both CLI and programmatic executions work identically now.

The original model store size was 144.4GB reduced to 22.2GB.

I believe this issue can be close.

Thanks,  
Borell