# Fine tune a saved model with custom tokenizer

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-a-saved-model-with-custom-tokenizer%2F2772%3E%5C
**Category:** 🤗Transformers\
**Created:** [December 10, 2020, 2:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-a-saved-model-with-custom-tokenizer%2F2772 "2020-12-10T14:02:33Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Adel](https://avatars.discourse-cdn.com/v4/letter/a/ec9cab/32.png) [@Adel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAdel)%5C
**Post date:** [December 10, 2020, 2:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-a-saved-model-with-custom-tokenizer%2F2772%2F1 "2020-12-10T14:02:33Z")

</div>

I am using a RoBERTa based model for pre-training and fine-tuning.

To pre-train, I use RobertaForMaskedLM with a **customized tokenizer**. This means I used my tokenizer in the LineByLineTextDataset() and pre-trained my model for masked language modeling.

However, for fine tuning, When I want to use my dataset with labels for a classification task, I think I must use my **customized tokenizer** before feeding my data to the model for fine tuning.

My question is, How can I use my tokenizer to prepare the data and fine tune my pre-trained model?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [December 11, 2020, 9:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-a-saved-model-with-custom-tokenizer%2F2772%2F2 "2020-12-11T09:58:38Z")

</div>

hi @Adel

You could save your custom tokenizer using the `save_pretrained`  
method and then load it again using `from_pretrained` method. So for classification fine-tuning you could just use the custom tokenizer. And if you are using the official transformer examples script then all you need to do is, pass the tokenizer using the `--tokenizer_name_or_path` argument.

---

<div class="post-metadata">

**Author:** ![Adel](https://avatars.discourse-cdn.com/v4/letter/a/ec9cab/32.png) [@Adel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAdel)%5C
**Post date:** [December 11, 2020, 10:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-a-saved-model-with-custom-tokenizer%2F2772%2F3 "2020-12-11T10:46:40Z")

</div>

Thank you @valhalla, I used my tokenizer to tokenize my input texts before feeding them to my ’ RobertaForSequenceClassification’ model. Since in the documentation it is mentioned that the RoBERTa uses BPE tokenizer:

_[RoBERTa has the same architecture as BERT, but uses a byte-level BPE as a tokenizer (same as GPT-2) and uses a different pretraining scheme](https://huggingface.co/transformers/model_doc/roberta.html)._

I wonder if I use my custom tokenizer, the ‘RobertaForSequenceClassification’ still uses BPE and not using my custom tokenizer.

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [December 15, 2020, 12:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-a-saved-model-with-custom-tokenizer%2F2772%2F4 "2020-12-15T12:27:21Z")

</div>

What that means is the `RobertaTokenizer` uses `BPE` for `tokenization`. BPE is one of the tokenization method, using which we train tokenizer. Hope this makes it clear.