# Is model.generate slower than model forward call?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-model-generate-slower-than-model-forward-call%2F102864%3E%5C
**Category:** 🤗Transformers\
**Created:** [August 18, 2024, 3:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-model-generate-slower-than-model-forward-call%2F102864 "2024-08-18T15:25:21Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![geekifan](https://avatars.discourse-cdn.com/v4/letter/g/5daacb/32.png) [@geekifan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgeekifan)%5C
**Post date:** [August 18, 2024, 3:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-model-generate-slower-than-model-forward-call%2F102864%2F1 "2024-08-18T15:25:21Z")

</div>

I need to get the hidden states when the model outputs the next token. I compare model.generate with model forward and find that the speed of model forward call is faster than model.generate.

Example:  
Model: llava next  
Prompt: \<image\>\n Summarize it in one word:  
model forward call:

```auto
inputs = processor(input_prompts, images=batch['image'], return_tensors="pt", padding=True).to(device)
emb = model(**inputs, output_hidden_states=True, return_dict=True).hidden_states[-1][:, -1, :]

```

model.generate:

```auto
inputs = processor(input_prompts, images=batch['image'], return_tensors="pt", padding=True).to(device)
emb = model.generate(**inputs, max_new_tokens=60, output_hidden_states=True, return_dict_in_generate=True).hidden_states[0][-1][:, -1, :]

```

model forward call can process 1000 samples in 2 minutes but model.generate needs 20 minutes which is 10x slower than model forward call.

---

<div class="post-metadata">

**Author:** ![darknoon](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/darknoon/32/29233_2.png) [@darknoon](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdarknoon)%5C
**Post date:** [August 18, 2024, 6:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-model-generate-slower-than-model-forward-call%2F102864%2F2 "2024-08-18T18:38:34Z")

</div>

Expected. `model.generate()` is running inference up to 60 times in a loop to do auto-regressive inference, whereas running the model can be easily parallelized. If you want to speed up generation, look into something like VLLM for inference.