# Slow Tokenizer adds whitespace after special token

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fslow-tokenizer-adds-whitespace-after-special-token%2F48966%3E%5C
**Category:** 🤗Transformers\
**Created:** [August 1, 2023, 10:37am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fslow-tokenizer-adds-whitespace-after-special-token%2F48966 "2023-08-01T10:37:49Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![wlhgtc](https://avatars.discourse-cdn.com/v4/letter/w/7ba0ec/32.png) [@wlhgtc](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwlhgtc)%5C
**Post date:** [August 1, 2023, 10:37am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fslow-tokenizer-adds-whitespace-after-special-token%2F48966%2F1 "2023-08-01T10:37:49Z")

</div>

@ArthurZ  
Sorry I misssed some information, but I’m still confused after reading your reply at [Slow Tokenizer adds whitespace after special token · Issue #25073 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/25073#issuecomment-1655379486).  
As mentioned in [https://github.com/huggingface/transformers/pull/24565#issuecomment-1656719744](https://github.com/huggingface/transformers/pull/24565#issuecomment-1656719744)  
**How could a word that comes after a special token be considered part of that special token?**

Suppose we add a special token `<bot>` into `LlamaTokenizer`.

```python
    txt = "<bot>" + "How are you" 
    tokenizer1 = LlamaTokenizer.from_pretrained(
        "./resources/models/llama-2-7b-hf", legacy=True, use_fast=False
    )
    tokenizer2 = LlamaTokenizer.from_pretrained(
        "./resources/models/llama-2-7b-hf", legacy=False, use_fast=False
    )

    t1 = tokenizer1.tokenize(txt)
    t2 = tokenizer2.tokenize(txt)

```

The result is as follws:

```python
t1:['<bot>','▁How', '▁are', '▁you']
t2:['<bot>', 'How', '▁are', '▁you']

```

If we consider `<bot>` and `How` to be part of the same word, `tokenizer1` performs correctly. However, if we consider them to be two different words, then `tokenizer2` would be better. Although the original text is `<bot>How`, I don’t think it’s a legitimate word. Hence, `tokenizer2` seems to correct it."

If there are any inaccuracies about my points, please feel free to correct them.

---

<div class="post-metadata">

**Author:** ![ArthurZ](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/arthurz/32/26972_2.png) [@ArthurZ](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FArthurZ)%5C
**Post date:** [August 2, 2023, 9:32am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fslow-tokenizer-adds-whitespace-after-special-token%2F48966%2F2 "2023-08-02T09:32:07Z")

</div>

Hey! Glad you pinged me here 😉 !  
So I totally agree with you, they are different words. I don’t know why your question implies that I meant that a word should be part of a special token, but no indeed it is not.

Even if we consider `<bot>` and `How` to be part of the same word, `['<bot>','▁How']` is still wrong. The `'▁'` token is not there to split words, it’s a space. So for example `'▁are'` is a different token from `are`. `'▁are'` is a beginning of a word, ( meaning there is always a space before it) while `'are'` is in a word like `'care'` or `'aware'` etc (depending on the merges and vocab).

---

<div class="post-metadata">

**Author:** ![wlhgtc](https://avatars.discourse-cdn.com/v4/letter/w/7ba0ec/32.png) [@wlhgtc](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwlhgtc)%5C
**Post date:** [August 2, 2023, 10:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fslow-tokenizer-adds-whitespace-after-special-token%2F48966%2F3 "2023-08-02T10:27:15Z")

</div>

In the discussion related to my comment in [#25073](https://github.com/huggingface/transformers/issues/25073#issuecomment-1655271420), we’ve observed distinct behaviors in the case of words like `<bot>traditionally` when setting `legacy` to different values.

```auto
# when setting it to True
['<bot>', '▁tradition', 'ally']
# when setting it to False
['<bot>', 'tradition', 'ally']

```

The setting `True` adds an unintended space between `<bot>` and `tradition` , possibly due to a mistake or bug. However, it does correct the spelling as these two parts are not typically perceived as a single word. It’s likely a typographical error, and the intended input should be `<bot> traditionally` rather than `<bot>traditionally` . I initially thought of this as a peculiar “feature” that didn’t necessitate any changes.  
However, upon further consideration, encoding text such as `<bot>.`. should ideally tokenize to `['<bot>', '.']`. Without any modification, it could result in` ['<bot>', '▁.']`.

In conclusion, it’s crucial and worthy of our attention and effort.

Thanks again for your quickly reply and great work(about this PR).

---

<div class="post-metadata">

**Author:** ![ArthurZ](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/arthurz/32/26972_2.png) [@ArthurZ](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FArthurZ)%5C
**Post date:** [August 8, 2023, 10:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fslow-tokenizer-adds-whitespace-after-special-token%2F48966%2F4 "2023-08-08T10:25:15Z")

</div>

Could you reming me why you need `legacy=True`?  
Because

```python
# when setting it to False
['<bot>', 'tradition', 'ally']

```

seems correct no?  
If you have an input that is `<bot> Hey` and you want to force this to encode to `<bot>Hey` either removing prefix space, or setting `rstrip` to `True` could fix this no?

---

<div class="post-metadata">

**Author:** ![wlhgtc](https://avatars.discourse-cdn.com/v4/letter/w/7ba0ec/32.png) [@wlhgtc](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwlhgtc)%5C
**Post date:** [August 8, 2023, 11:02am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fslow-tokenizer-adds-whitespace-after-special-token%2F48966%2F5 "2023-08-08T11:02:44Z")

</div>

When setting `legacy=True`: If I forgot to add a space between `<bot>` and `traditionally` , It can be fixed by adding an “extra” space between them.

```python
['<bot>', '▁tradition', 'ally']

```

The premise here is that the token after the special token is treated as an individual token, even though we don’t add a space before it.