# Maybe there is a bug in BertTokenizer?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmaybe-there-is-a-bug-in-berttokenizer%2F4671%3E%5C
**Category:** 🤗Transformers\
**Created:** [March 19, 2021, 6:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmaybe-there-is-a-bug-in-berttokenizer%2F4671 "2021-03-19T06:00:16Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sniper](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sniper/32/578_2.png) [@Sniper](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSniper)%5C
**Post date:** [March 19, 2021, 6:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmaybe-there-is-a-bug-in-berttokenizer%2F4671%2F1 "2021-03-19T06:00:16Z")

</div>

I try to add a custom token in tokenizer.

I find this code in source code:

```
    Args:
        new_tokens (:obj:`List[str]`or :obj:`List[tokenizers.AddedToken]`):
            Token(s) to add in vocabulary. A token is only added if it's not already in the vocabulary (tested by
            checking if the tokenizer assign the index of the ``unk_token`` to them).
        special_tokens (:obj:`bool`, `optional`, defaults to :obj:`False`):
            Whether or not the tokens should be added as special tokens.

    Returns:
        :obj:`int`: The number of tokens actually added to the vocabulary.

    Examples::

        # Let's see how to increase the vocabulary of Bert model and tokenizer
        tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
        model = BertModel.from_pretrained('bert-base-uncased')

        num_added_toks = tokenizer.add_tokens(['new_tok1', 'my_new-tok2'])
        print('We have added', num_added_toks, 'tokens')
        # Note: resize_token_embeddings expects to receive the full size of the new vocabulary, i.e. the length of the tokenizer.
        model.resize_token_embeddings(len(tokenizer))
    """
    new_tokens = [str(tok) for tok in new_tokens]

```

but after I run the code  
`tokenizer.add_tokens('ss##e',special_tokens=True)`  
there is no change in special\_tokens

 ![图片](https://us1.discourse-cdn.com/hellohellohello/original/2X/0/037ddbeeea1ad5e5b9609e91fc88a843c4f9d18f.png)

I have tried for serval times  
 ![图片](https://us1.discourse-cdn.com/hellohellohello/original/2X/9/9e97265b46218291823daa5fee2198c98767a95a.png)

but it seem like is the same change weather True or False.

I notice that there is a special explain for albert.

```
    # Make sure we don't split on any special tokens (even they were already in the vocab before e.g. for Albert)
    if special_tokens:
        self.unique_no_split_tokens = sorted(set(self.unique_no_split_tokens).union(set(new_tokens)))
    else:
        # Or on the newly added tokens
        self.unique_no_split_tokens = sorted(set(self.unique_no_split_tokens).union(set(tokens_to_add)))

```

Is there a different between bert and albert that I don’t know? or there is someting wrong.