# Best practices for improving text generation speed?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbest-practices-for-improving-text-generation-speed%2F17367%3E%5C
**Category:** Beginners\
**Created:** [April 30, 2022, 9:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbest-practices-for-improving-text-generation-speed%2F17367 "2022-04-30T21:59:51Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![ivnle](https://avatars.discourse-cdn.com/v4/letter/i/b5a626/32.png) [@ivnle](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fivnle)%5C
**Post date:** [April 30, 2022, 9:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbest-practices-for-improving-text-generation-speed%2F17367%2F1 "2022-04-30T21:59:52Z")

</div>

Aside from buying a faster/larger GPU, are there any best practices for increasing text generation speed? Specifically, I’d like to use GPT-2 (of various sizes) to generate a large set of text (5,000 examples, 1,000 BPE tokens each).

My initial research shows there are a few options:  
(a) [DeepSpeed for inference](https://www.deepspeed.ai/tutorials/inference-tutorial/)  
(b) [batched generation](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbatch-generation-with-gpt2%2F1517)  
(c) [fp16 inference](https://huggingface.co/docs/transformers/performance#fp16-inference). Although I’m not sure how to do this outside of Trainer. I could call `model.half()` but it’s not clear to me if that’s the right way to go about this.

Any advice is appreciated!