# Streaming token output from models like T5
**URL:**
**Author:**  [@zhuda](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzhuda)%5C
**Post date:** [January 31, 2023, 3:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-token-output-from-models-like-t5%2F30785%2F1 "2023-01-31T03:09:59Z")
Hey, I’m looking to stream tokens out of text2textgeneration models (more specifically those from the T5 family). Similar to what you see in OpenAI playground when requesting generations.
For text-generation models I’ve been able to emulate this (since we’re just predicting current token from the ones before it) by setting a max\_time parameter and appending model generations to the prompt to continuously call until I’ve reached the number of tokens I wanted to generate. But I can’t do the same for text2text models.
Any advice on how I could stream output from models like T5? Is it even possible with the architecture?
---
**Author:**  [@zhuda](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzhuda)%5C
**Post date:** [February 3, 2023, 10:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-token-output-from-models-like-t5%2F30785%2F2 "2023-02-03T22:46:56Z")
Got a solution working, in `generate()` for the different types of sampling for example `greedy_search()` there is a `next_token` variable you can incrementally get the subsequent tokens generated by the model as soon as they are done. You’ll have to decode it yourself and encode the special rules you’d get from `decode()` but it works well. Monkey patched it with a new `greedy_search()` yielding `next_token` on each generation. Hope this helps for anyone looking to do the same.
---
**Author:**  [@peakji](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpeakji)%5C
**Post date:** [March 7, 2023, 2:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-token-output-from-models-like-t5%2F30785%2F3 "2023-03-07T14:42:21Z")
Inspired by the solution form @zhuda , I made a streaming generation service for Hugging Face [transformers](https://github.com/huggingface/transformers) that is fully compatible with the OpenAI API: [https://github.com/hyperonym/basaran](https://github.com/hyperonym/basaran)
---
**Author:**  [@vblagoje](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvblagoje)%5C
**Post date:** [March 14, 2023, 8:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-token-output-from-models-like-t5%2F30785%2F4 "2023-03-14T08:40:02Z")
Hey guys, why didn’t you register one of the callbacks in generate and choose instead to monkey patch?
---
**Author:**  [@zhuda](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzhuda)%5C
**Post date:** [March 14, 2023, 11:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-token-output-from-models-like-t5%2F30785%2F5 "2023-03-14T23:20:57Z")
This is awesome, can’t wait take a proper look!
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [March 27, 2023, 7:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-token-output-from-models-like-t5%2F30785%2F6 "2023-03-27T07:11:41Z")
Hi,
We recently released a bunch of open-source tools to do this with any HuggingFace model.
Check out our new library “text-generation-inference”: [GitHub - huggingface/text-generation-inference: Large Language Model Text Generation Inference](https://github.com/huggingface/text-generation-inference). It powers this Space: [Chat Llm Streaming - a Hugging Face Space by olivierdehaene](https://huggingface.co/spaces/olivierdehaene/chat-llm-streaming).
cc @olivierdehaene
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [June 5, 2023, 11:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-token-output-from-models-like-t5%2F30785%2F7 "2023-06-05T11:10:05Z")
Update:
We now also support a new `Streamer` class that works in tandem with the `generate` method.
Here’s a great Twitter thread by @joaogante going over it: [https://twitter.com/joao\_gante/status/1643330507093196800](https://twitter.com/joao_gante/status/1643330507093196800)
---
**Author:**  [@zhuda](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzhuda)%5C
**Post date:** [June 7, 2023, 10:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-token-output-from-models-like-t5%2F30785%2F8 "2023-06-07T22:27:13Z")
This is perfect, will port over to this solution soon for OpenPlayground