# Efficient detokenization method

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fefficient-detokenization-method%2F3421%3E%5C
**Category:** 🤗Transformers\
**Created:** [January 28, 2021, 6:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fefficient-detokenization-method%2F3421 "2021-01-28T18:29:49Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![berkayberabi](https://avatars.discourse-cdn.com/v4/letter/b/258eb7/32.png) [@berkayberabi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fberkayberabi)%5C
**Post date:** [January 28, 2021, 6:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fefficient-detokenization-method%2F3421%2F1 "2021-01-28T18:29:50Z")

</div>

Tokenizer has **call** function nad it accepts List[List[string]] so one can tokenize multiple samples at once efficiently. I am looking for a function that reverts this behavior. There are convert\_ids\_to\_tokens() and convert\_tokens\_to\_string() but they only accept List[int] and List[string]. I do not want to iterate over the sample and convert them back one by one.

Is there an efficient method for this? I am using T5

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [January 28, 2021, 8:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fefficient-detokenization-method%2F3421%2F2 "2021-01-28T20:25:28Z")

</div>

There is the `decode` method that could help.

---

<div class="post-metadata">

**Author:** ![berkayberabi](https://avatars.discourse-cdn.com/v4/letter/b/258eb7/32.png) [@berkayberabi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fberkayberabi)%5C
**Post date:** [January 28, 2021, 10:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fefficient-detokenization-method%2F3421%2F3 "2021-01-28T22:01:55Z")

</div>

are you sure that it can take multiple sentences? from the documentation, it seems to me that it only accepts one sample.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [January 28, 2021, 10:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fefficient-detokenization-method%2F3421%2F4 "2021-01-28T22:04:00Z")

</div>

For multiple sentences there is its counterpart `batch_decode`.
