>
> Hi,
> I trained my own tokenizer using tokenizers library and want to use vocab …file with Fast Tokenizers.
> However, fast tokenizer split special tokens like below even \`PreTrainedTokenizerFast\` have special\_tokens.
>
> \`\`\`python
> \>\>\> tokenizer = ElectraTokenizerFast('my\_vocab.txt', do\_lower\_case=False, model\_max\_length=512)
> \>\>\> tokenizer
> PreTrainedTokenizerFast(name\_or\_path='',
> vocab\_size=32000,
> model\_max\_len=512,
> is\_fast=True,
> padding\_side='right',
> special\_tokens={'unk\_token': '\[UNK\]', 'sep\_token': '\[SEP\]', 'pad\_token': '\[PAD\]', 'cls\_token': '\[CLS\]', 'mask\_token': '\[MASK\]'})
> \>\>\> tokenizer.tokenize("\[MASK\] is capital of France.", add\_special\_tokens=True)
> \['\[CLS\]', '\[', 'MA', '##SK', '\]', 'is', 'cap', '##ital', 'of', 'France', '.', '\[SEP\]'\]
> \`\`\`
>
> but when I \`add\_special\_tokens\` manually, it works well
>
> \`\`\`python
> \>\>\> tokenizer = ElectraTokenizerFast('my\_vocab.txt', do\_lower\_case=False, model\_max\_length=512)
> \>\>\> special\_tokens = {
> "unk\_token": "\[UNK\]",
> "sep\_token": "\[SEP\]",
> "pad\_token": "\[PAD\]",
> "cls\_token": "\[CLS\]",
> "mask\_token": "\[MASK\]" }
> \>\>\> tokenizer.add\_special\_tokens(special\_tokens)
> \>\>\> tokenizer.tokenize("\[MASK\] is capital of France.", add\_special\_tokens=True)
> \['\[CLS\]', '\[MASK\]', 'is', 'cap', '##ital', 'of', 'France', '.', '\[SEP\]'\]
> \`\`\`
>
>
> Should I add special tokens again with \`add\_special\_tokens\` manually, even if \`PreTrainedTokenizerFast\` have default special tokens?
---
**Author:**  [@torch-nn](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftorch-nn)%5C
**Post date:** [June 30, 2025, 3:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-is-splitting-special-token%2F161147%2F3 "2025-06-30T03:15:56Z")
I just tried `add_tokens` (like below) and its still not working:
```auto
wrapped_tokenizer = PreTrainedTokenizerFast(
tokenizer_object=tokenizer,
unk_token="[UNK]",
cls_token="[CLS]",
sep_token="[SEP]",
pad_token="[PAD]",
mask_token="[MASK]",
)
wrapped_tokenizer.add_tokens(['HTTPURL', '@USER'])
```
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [June 30, 2025, 5:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-is-splitting-special-token%2F161147%2F4 "2025-06-30T05:29:37Z")
Hmm… How about `save_pretrained` with `from_pretrained` way?
> [@Cannot create an identical PretrainedTokenizerFast object from a Tokenizer created by tokenizers library](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcannot-create-an-identical-pretrainedtokenizerfast-object-from-a-tokenizer-created-by-tokenizers-library%2F9317)%3A
>
> I can’t seem to create a “PreTrainedTokenizerFast” object from my original tokenizers tokenizer object that has the same proporties. This is the code for a byte pair tokenizer I have experimented on. The resulting fast tokenizer does not have a [PAD] token, and does not have any special tokens at all. tokenizer = ByteLevelBPETokenizer() tokenizer.preprocessor = pre\_tokenizers.BertPreTokenizer() tokenizer.normalizer = normalizers.BertNormalizer() tokenizer.train\_from\_iterator(doc…
>
>
> I can't seem to create a "PreTrainedTokenizerFast" object from my original \`toke…nizers\` tokenizer object that has the same proporties. This is the code for a byte pair tokenizer I have experimented on. The resulting fast tokenizer does not have a \[PAD\] token, and does not have any special tokens at all.
> \`\`\`
> tokenizer = ByteLevelBPETokenizer()
> tokenizer.preprocessor = pre\_tokenizers.BertPreTokenizer()
> tokenizer.normalizer = normalizers.BertNormalizer()
> tokenizer.train\_from\_iterator(docs, vocab\_size=16\_000, min\_frequency=15, special\_tokens = \["\[UNK\]", "\[CLS\]", "\[SEP\]", "\[PAD\]", "\[MASK\]"\])
> tokenizer.\_tokenizer.post\_processor = processors.BertProcessing(
> ("\[SEP\]", tokenizer.token\_to\_id("\[SEP\]")),
> ("\[CLS\]", tokenizer.token\_to\_id("\[CLS\]")),
> )
> tokenizer.enable\_truncation(max\_length=256)
> tokenizer.enable\_padding(pad\_id=3, pad\_token="\[PAD\]")
> fast\_tokenizer = PreTrainedTokenizerFast(tokenizer\_object=tokenizer)
> \`\`\`
>
> The result of printing the fast\_tokenizer is:
> \`\`\`
> PreTrainedTokenizerFast(name\_or\_path='', vocab\_size=16000, model\_max\_len=1000000000000000019884624838656, is\_fast=True, padding\_side='right', special\_tokens={})
> \`\`\`
> Which \`model\_max\_len\` and \`special\_tokens\` are wrong in it. Also, there is no pad\_token and pad\_token\_id in the fast\_tokenizer object. (the warning for pad\_token for example: \`Using pad\_token, but it is not set yet.\`) Have I done anything wrong, or is this not supposed to happen?
>
> The versions of libraries I'm using:
> \`\`\`
> \['tokenizers 0.10.3',
> 'transformers 4.10.0.dev0'\]
> \`\`\`
>
>
> It works when using OpenAIGPTTokenizer additional\_special\_tokens parameter. It's clear that after declare additional\_special\_tokens parameter, OpenAIGPTTokenizer tokenize \
> \
>
>
>
>
> The version of transformers is 4.2.2
> And I'm not sure this problem is related with \[issue624\](https://github.com/google-research/text-to-text-transfer-transformer/issues/624) in T5 which talk about SentencePiece extra vocab.
>
> Thank you for your feedback
>