# Is detokenize available in transformer lib?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-detokenize-available-in-transformer-lib%2F34620%3E%5C
**Category:** 🤗Transformers\
**Created:** [March 25, 2023, 6:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-detokenize-available-in-transformer-lib%2F34620 "2023-03-25T06:24:27Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![innat](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/innat/32/33255_2.png) [@innat](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Finnat)%5C
**Post date:** [March 25, 2023, 6:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-detokenize-available-in-transformer-lib%2F34620%2F1 "2023-03-25T06:24:27Z")

</div>

I’ve searched on doc but couldn’t find any hint.

Generally, `detokenize` is the inverse of the `tokenize` method, and can be used to reconstrct a string from a set of tokens.

```python
from transformers import TFBertTokenizer

tf_tokenizer = TFBertTokenizer.from_pretrained("bert-base-uncased")

# something like
tf_tokenizer.encode([string]) # o/p: ids / token
tf_tokenizer.decode([1,2,3]) # o/p: string

```

* * *

Available in some extent

> <https://github.com/keras-team/keras-nlp/blob/b35e83fbc1/keras_nlp/models/bert/bert_tokenizer_test.py>

> <https://github.com/tensorflow/text/blob/dc5983e1be3140130f4eb70e981c7cb0fecadd85/tensorflow_text/python/ops/tokenization.py#L179>

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [March 27, 2023, 12:40pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-detokenize-available-in-transformer-lib%2F34620%2F2 "2023-03-27T12:40:29Z")

</div>

cc @Rocketknight1

---

<div class="post-metadata">

**Author:** ![Rocketknight1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/rocketknight1/32/21055_2.png) [@Rocketknight1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRocketknight1)%5C
**Post date:** [April 24, 2023, 1:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-detokenize-available-in-transformer-lib%2F34620%2F3 "2023-04-24T13:09:32Z")

</div>

Hi @innat, and sorry for the delay! I don’t think our TF in-graph tokenizers support decoding/detokenization. However, our main tokenizers do. So you could do something like

```auto
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

tokenizer.decode([1, 2, 3])

```

This should work for most purposes - do you have a usecase for wanting to do **de** tokenization inside a TF graph? We’re very interested if so, because we assumed people would generally not need that!