# Creating a custom tokenizer for Roberta

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcreating-a-custom-tokenizer-for-roberta%2F2809%3E%5C
**Category:** Beginners\
**Created:** [December 14, 2020, 12:31am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcreating-a-custom-tokenizer-for-roberta%2F2809 "2020-12-14T00:31:35Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![david-waterworth](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@david-waterworth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdavid-waterworth)%5C
**Post date:** [December 14, 2020, 12:31am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcreating-a-custom-tokenizer-for-roberta%2F2809%2F1 "2020-12-14T00:31:35Z")

</div>

RobertaTokenizerFast seems to be ignoring my Lowercase() normaliser. I’ve created a custom tokeniser as follows:

```
tokenizer = Tokenizer(BPE(unk_token="<unk>", end_of_word_suffix="</w>"))
tokenizer.normalizer = Lowercase()
tokenizer.pre_tokenizer = Sequence([Whitespace(), Digits(individual_digits=False), Punctuation()])

trainer = BpeTrainer(
    vocab_size=3000,
    special_tokens=["<s>", "</s>", "<unk>", "<pad>", "<mask>"]
)
tokenizer.train(trainer, files)
tokenizer.post_processor = RobertaProcessing(
    cls=("<s>", tokenizer.token_to_id("<s>")),
    sep=("</s>", tokenizer.token_to_id("</s>")),
)
tokenizer.decoder = BPEDecoder(suffix="</w>")

```

(I’m not 100% sure if the BPE suffix is required?)

I then save as

```
tokenizer.model.save("./models/roberta")
tokenizer.save("./models/roberta/config.json") 

```

The reason I need a custom tokeniser is my examples aren’t white-space delimited, i…e

```
tokenizer.encode("AHU-01-SAT").tokens

```

> [‘\<s\>’, ‘ahu’, ‘-’, ‘01’, ‘-’, ‘sat’, ‘\</s\>’]

The following doesn’t return the correct tokens:

```
from transformers import RobertaTokenizerFast  
tokenizer = RobertaTokenizerFast.from_pretrained(".models/roberta", max_len=512)  
tokenizer("AHU-01-SAT")

```

> {‘input\_ids’: [0, 40, 112, 40, 1], ‘attention\_mask’: [1, 1, 1, 1, 1]}

It’s missing the first and last tokens (plus it’s not even replacing them with “”?

If I manually apply the normalisation I get the correct tokenisation

```
tokenizer("ahu-01-sat")

```

{‘input\_ids’: [0, 109, 40, 112, 40, 598, 1], ‘attention\_mask’: [1, 1, 1, 1, 1, 1, 1]}

I tried AutoTokenizer and observed the same issue - am I doing something wrong?

---

<div class="post-metadata">

**Author:** ![david-waterworth](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@david-waterworth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdavid-waterworth)%5C
**Post date:** [December 14, 2020, 12:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcreating-a-custom-tokenizer-for-roberta%2F2809%2F2 "2020-12-14T00:56:16Z")

</div>

Ahh, figured it out, it should be

```
tokenizer.save("./models/roberta/tokenizer.json") 

```

not

```
tokenizer.save("./models/roberta/config.json")
```

---

<div class="post-metadata">

**Author:** ![david-waterworth](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@david-waterworth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdavid-waterworth)%5C
**Post date:** [December 14, 2020, 10:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcreating-a-custom-tokenizer-for-roberta%2F2809%2F3 "2020-12-14T10:05:30Z")

</div>

One question I still have though is what’s the difference between `tokenizer.json` and `tokenizer_config.json` - for example if I use `RobertaTokenizerFast.from_pretrained()` I get the following.

> Didn’t find file added\_tokens.json. We won’t load it.  
> Didn’t find file special\_tokens\_map.json. We won’t load it.  
> Didn’t find file tokenizer\_config.json. We won’t load it.  
> loading file vocab.json  
> loading file merges.txt  
> loading file tokenizer.json

And If I try and load this tokeniser with AutoTokenizer.from\_pretrained() it fails since there’s no `config.json` - I’m trying to train from scratch using the `run_mlm.py` script.

Is there a step by step guide to training a (custom) tokenizer, then a model (from scratch) using `run_mlm.py`?

---

<div class="post-metadata">

**Author:** ![ninjalu](https://avatars.discourse-cdn.com/v4/letter/n/c4cdca/32.png) [@ninjalu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fninjalu)%5C
**Post date:** [August 1, 2021, 8:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcreating-a-custom-tokenizer-for-roberta%2F2809%2F4 "2021-08-01T20:59:25Z")

</div>

@david-waterworth Have you found your answer? I am following the code [here](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb#scrollTo=4keFBUjQFOD1) to train a new language model. I am getting config.json not found when trying to load the pertained tokenizer. :(. Very puzzled, not sure what I did wrong?

---

<div class="post-metadata">

**Author:** ![david-waterworth](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@david-waterworth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdavid-waterworth)%5C
**Post date:** [August 1, 2021, 11:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcreating-a-custom-tokenizer-for-roberta%2F2809%2F5 "2021-08-01T23:06:54Z")

</div>

I added a config.json file manually in the same folder as the tokeniser files, i.e.

{  
“model\_type”: “roberta”,  
“vocab\_size”: 52000,  
“type\_vocab\_size”: 1  
}

I’m not sure there’s a step-by-step guide but in general if you AutoTokenizer.from\_pretrained() rather than RobertaTokenizerFast.from\_pretrained() then you need a config.json containing at least the model\_type so Auto knowns which model family you’re using. Plus any other parameters that differ to the roberta defaults (such as the vocab size).

You need to do the same thing for your model, i.e. in the mlm script it loads AutoConfig.from\_pretrained(config\_name) to initialise the actual model parameters used in training (number of heads etc). You need to create your own config.json containing the parameters from RobertaConfig so AutoConfig can load them (best thing to do is start by copying the config.json for Roberta from the model hub then modify as required).

---

<div class="post-metadata">

**Author:** ![david-waterworth](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@david-waterworth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdavid-waterworth)%5C
**Post date:** [August 1, 2021, 11:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcreating-a-custom-tokenizer-for-roberta%2F2809%2F6 "2021-08-01T23:19:19Z")

</div>

Also note you you create a config file from a config class using save\_pretrained, i.e. soemthing like (I’ve not tested this)

config.save\_pretrained("./EsperBERTo/config.json")

should work and seems to be missing from the example you’re following