# ONNX Flan-T5 Model OOM on GPU **URL:** **Author:** ![eusip](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/eusip/32/2714_2.png) [@eusip](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Feusip)%5C **Post date:** [April 12, 2023, 1:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fonnx-flan-t5-model-oom-on-gpu%2F36342%2F1 "2023-04-12T13:31:11Z") I have finetuned a private variant of Flan-T5 XXL and exported it locally using Optimum. `python -m transformers.onnx --model=model_id --feature=seq2seq-lm-with-past --framework=pt .` Executing this command has generated the following files: - config.json - decoder\_model.onnx\_data (25GB) - decoder\_with\_past\_model.onnx\_data (22GB) - encoder\_model.onnx\_data (18GB) - spiece.model - tokenizer\_config.json - decoder\_model.onnx - decoder\_with\_past\_model.onnx - encoder\_model.onnx - special\_tokens\_map.json - tokenizer.json So far so good. I am now trying to load this model onto a GPU using provider=‘CUDAExecutionProvider’ and run out of memory on an A100 (80GB). I have tried creating inference session for encoder\_model.onnx, decoder\_model.onnx and decoder\_with\_past\_model.onnx. ```auto encoder_sess = InferenceSession("/mnt/training/mirage-onnx/no_opt/encoder_model.onnx", providers=['CUDAExecutionProvider']) decoder_sess = InferenceSession("/mnt/training/mirage-onnx/no_opt/decoder_model.onnx", providers=['CUDAExecutionProvider']) decoder_with_past_sess = InferenceSession("/mnt/training/mirage-onnx/no_opt/decoder_with_past_model.onnx", providers=['CUDAExecutionProvider']) ``` encoder\_sess and decoder\_sess are created with no problem. decoder\_with\_past\_sess results in the following error: ```auto Traceback (most recent call last): File "generate-mirage-onnx-alt.py", line 58, in decoder_with_past_sess = InferenceSession("/mnt/training/mirage-onnx/no_opt/decoder_with_past_model.onnx", providers=['CUDAExecutionProvider']) File "/opt/conda/envs/accelerate/lib/python3.7/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 347, in __init__ self._create_inference_session(providers, provider_options, disabled_optimizers) File "/opt/conda/envs/accelerate/lib/python3.7/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 395, in _create_inference_session sess.initialize_session(providers, provider_options, disabled_optimizers) onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: /onnxruntime_src/onnxruntime/core/framework/bfc_arena.cc:342 void* onnxruntime::BFCArena::AllocateRawInternal(size_t, bool) Failed to allocate memory for requested buffer of size 167772160 ``` Thus far I have not run any optimizations during the export of the model to ONNX. Nonetheless the run away memory usage of decoder\_with\_past\_model.onnx is rather strange. @echarlaix any tips or suggestions would be greatly appreciated. ---
**Author:** ![eusip](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/eusip/32/2714_2.png) [@eusip](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Feusip)%5C **Post date:** [April 13, 2023, 10:49am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fonnx-flan-t5-model-oom-on-gpu%2F36342%2F2 "2023-04-13T10:49:18Z")
In the end I found out that I was not encountering a bug and my GPU was truly OOM. Novice misunderstanding as I am new to ONNX. For those who are new to ONNX the memory requirements are actually 4-5x that of running a model in the PyTorch format. ---
**Author:** ![fxmarty](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fxmarty/32/23782_2.png) [@fxmarty](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffxmarty)%5C **Post date:** [June 15, 2023, 5:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fonnx-flan-t5-model-oom-on-gpu%2F36342%2F3 "2023-06-15T05:38:39Z")
@eusip Hi, part of the issue is that a few months ago we were using two separate .onnx for the decoder without/with past, effectively doubling the memory requirement for it. If you try to reexport your model on the latest optimum version with `optimum-cli export onnx`, you should find a `decoder_model_merged.onnx` that is automatically used in `ORTModelForCausalLM`/`ORTModelForSeq2SeqLM`. Hopefully it can help a bit with memory usage. But it is true that, overall, ONNX Runtime is a pain when it comes to memory (especially on GPU), see for example: [[Performance] Find out why the GPU memory allocated with `CUDAExecutionProvider` is much larger than the ONNX size · Issue #14526 · microsoft/onnxruntime · GitHub](https://github.com/microsoft/onnxruntime/issues/14526)