# GPU inference slows down if done in a loop

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpu-inference-slows-down-if-done-in-a-loop%2F361%3E%5C
**Category:** 🤗Transformers\
**Created:** [July 17, 2020, 5:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpu-inference-slows-down-if-done-in-a-loop%2F361 "2020-07-17T17:00:39Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![issafuad](https://avatars.discourse-cdn.com/v4/letter/i/8c91f0/32.png) [@issafuad](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fissafuad)%5C
**Post date:** [July 17, 2020, 5:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpu-inference-slows-down-if-done-in-a-loop%2F361%2F1 "2020-07-17T17:00:39Z")

</div>

Hi I have noticed that inference time is very quick if running the model on one batch. However, once inference is ran in a loop - even if on the same input - it slows down significantly.

I have actually seen the same behaviour on tensorflow models. Is an expected behaviour or is an issue with cuda etc.

Please find the notebook to see the issue  
[https://colab.research.google.com/drive/1gqSzQqFm8HL0OwmJzSRlcRFQ3FOpnvFh?usp=sharing](https://colab.research.google.com/drive/1gqSzQqFm8HL0OwmJzSRlcRFQ3FOpnvFh?usp=sharing)

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [July 20, 2020, 1:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpu-inference-slows-down-if-done-in-a-loop%2F361%2F2 "2020-07-20T13:42:41Z")

</div>

This is because Python is a slow language. You generally want to avoid a loop in Python to get performance, and want to use inputs in a batch to use the full performance of your hardware.