# BOS tokens for mBERT tokenizer

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbos-tokens-for-mbert-tokenizer%2F5467%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [April 13, 2021, 2:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbos-tokens-for-mbert-tokenizer%2F5467 "2021-04-13T14:49:50Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![rahular](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/rahular/32/1031_2.png) [@rahular](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frahular)%5C
**Post date:** [April 13, 2021, 2:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbos-tokens-for-mbert-tokenizer%2F5467%2F1 "2021-04-13T14:49:50Z")

</div>

Default mBERT tokenizes a sentence as `['[CLS]', 'This', 'is', 'a', 'sample', 'sentence', '[SEP]']`. I want to change this behaviour and add a language specific token after the `CLS` token like this: `['[CLS]', ' __en__', 'This', 'is', 'a', 'sample', 'sentence', '[SEP]']`

I know `TemplateProcessing` can be used to achieve this if the language token doesn’t change

```auto
from tokenizers.processors import TemplateProcessing
tokenizer._tokenizer.post_processor = TemplateProcessing(
    single=f"{_lang_token} $A [SEP]",
    pair=f"{_lang_token} $A [SEP] $B:1 [SEP]:1",
    special_tokens=[("[SEP]", tokenizer.convert_tokens_to_ids("[SEP]")), 
                    (_lang_token, tokenizer.convert_tokens_to_ids(_lang_token))],
)

```

But in my case, the language token changes with every batch. What is the best way to add these tokens? Creating `TemplateProcessing` objects every time seems inefficient.

---

<div class="post-metadata">

**Author:** ![rahular](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/rahular/32/1031_2.png) [@rahular](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frahular)%5C
**Post date:** [April 14, 2021, 10:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbos-tokens-for-mbert-tokenizer%2F5467%2F2 "2021-04-14T10:13:07Z")

</div>

@sgugger any suggestions?