# Decode token IDs into a list (not a single string)

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecode-token-ids-into-a-list-not-a-single-string%2F42991%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [June 12, 2023, 10:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecode-token-ids-into-a-list-not-a-single-string%2F42991 "2023-06-12T22:58:16Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![steventrouble](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/steventrouble/32/16596_2.png) [@steventrouble](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsteventrouble)%5C
**Post date:** [June 12, 2023, 10:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecode-token-ids-into-a-list-not-a-single-string%2F42991%2F1 "2023-06-12T22:58:16Z")

</div>

`tokenizer.convert_ids_to_tokens` returns:

```auto
['ĠDrive', 'Ġwas', 'Ġhad', 'Ġwalked', "'s", ',', 'Ġlooked', ...]

```

I need the tokens without the special characters. `decode` does **not** work, because it only returns a single string.

Is there a function that outputs the plain tokens as a list?

---

<div class="post-metadata">

**Author:** ![ArthurZ](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/arthurz/32/26972_2.png) [@ArthurZ](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FArthurZ)%5C
**Post date:** [June 22, 2023, 7:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecode-token-ids-into-a-list-not-a-single-string%2F42991%2F2 "2023-06-22T07:11:37Z")

</div>

Hey! Not sure I completely understand, but the tokens that you have here are the `plain` tokens, as they are in the vocab / merge. You should modify the tokenizer if you do not want it to add the `spiece` token at the beginning. Which tokenizer are you using?

---

<div class="post-metadata">

**Author:** ![steventrouble](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/steventrouble/32/16596_2.png) [@steventrouble](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsteventrouble)%5C
**Post date:** [June 23, 2023, 3:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecode-token-ids-into-a-list-not-a-single-string%2F42991%2F3 "2023-06-23T03:40:18Z")

</div>

Thanks for the ping!

I was using the GPT byte level tokenizer.

I’m not sure if this is a hack, but to get the behavior I wanted, I just passed the token ids into `decode_batch` instead, and that returned each token without the odd encoding.

---

<div class="post-metadata">

**Author:** ![ArthurZ](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/arthurz/32/26972_2.png) [@ArthurZ](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FArthurZ)%5C
**Post date:** [September 18, 2023, 9:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecode-token-ids-into-a-list-not-a-single-string%2F42991%2F4 "2023-09-18T21:17:43Z")

</div>

It’s not a hack, but something I wish to improve! IMO `batch_decode` and `decode` should be merged into one as we only have `encode`

---

<div class="post-metadata">

**Author:** ![lone17](https://avatars.discourse-cdn.com/v4/letter/l/ccd318/32.png) [@lone17](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flone17)%5C
**Post date:** [March 11, 2025, 8:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecode-token-ids-into-a-list-not-a-single-string%2F42991%2F5 "2025-03-11T20:53:56Z")

</div>

Wow thank you ! Faced this today and this “hack” saved me. Btw after 2 years it’s still just a “hack” haha