# Regular tokens vs special tokens
**URL:**
**Author:**  [@Felipehonorato](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FFelipehonorato)%5C
**Post date:** [May 14, 2021, 7:40pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fregular-tokens-vs-special-tokens%2F6187%2F1 "2021-05-14T19:40:02Z")
Based on the CTRL approach on GPT2, i’m trying to add tokens in order to control my text generation style. Is there a difference between adding a token as a regular one and adding it as a special token?
---
**Author:**  [@lewtun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flewtun)%5C
**Post date:** [May 17, 2021, 10:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fregular-tokens-vs-special-tokens%2F6187%2F2 "2021-05-17T10:09:44Z")
hey @Felipehonorato, as far as i know, special tokens won’t be split by the tokenizer which might be handy in your case where you’re trying to incorporate control tokens. you can find more information in the docs [here](https://huggingface.co/transformers/internal/tokenization_utils.html?highlight=additional_special_tokens#transformers.tokenization_utils_base.SpecialTokensMixin.add_tokens).
---
**Author:**  [@Felipehonorato](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FFelipehonorato)%5C
**Post date:** [May 24, 2021, 10:51am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fregular-tokens-vs-special-tokens%2F6187%2F3 "2021-05-24T10:51:19Z")
Really thank you for the reply. Just to make sure i understood: When adding a word as a regular token, the tokenizer can maybe split this word into subwords, and when passing it as a special token this shouldn’t happen, am I right? I’ve made a couple of tests and by passing as special tokens worked pretty well
---
**Author:**  [@lewtun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flewtun)%5C
**Post date:** [May 26, 2021, 12:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fregular-tokens-vs-special-tokens%2F6187%2F4 "2021-05-26T12:20:40Z")
yes, you’re totally right regarding the difference between regular vs special tokens! in case you’re interested, this is all being handled by the `tokenizers` library, which has some extra details here: [Input sequences — tokenizers documentation](https://huggingface.co/docs/tokenizers/python/latest/api/reference.html?highlight=special#tokenizers.Tokenizer.add_special_tokens)
---
**Author:**  [@jaideepcs](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjaideepcs)%5C
**Post date:** [February 17, 2023, 5:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fregular-tokens-vs-special-tokens%2F6187%2F5 "2023-02-17T17:47:35Z")
@lewtun
if i have taxonomy of text
T1
T2
T3
to establish a context between T1 to T3 , would I need special token or normal tokens
T1 [L2] T2 [L3] T3
---
**Author:**  [@Mahdip72](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMahdip72)%5C
**Post date:** [January 8, 2024, 5:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fregular-tokens-vs-special-tokens%2F6187%2F6 "2024-01-08T05:30:45Z")
@lewtun Could you help us in this?
# [Tokenizer.add\_tokens automatically convert ESM2 new token to special](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-add-tokens-automatically-convert-esm2-new-token-to-special%2F67445)
We can not extend the embedding layer by adding new tokens using ESM-2 models.