# Custom tokenizer: finetune model or retrain model?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-tokenizer-finetune-model-or-retrain-model%2F76269%3E%5C
**Category:** 🤗Transformers\
**Created:** [March 6, 2024, 7:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-tokenizer-finetune-model-or-retrain-model%2F76269 "2024-03-06T19:51:55Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![edogarci](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/edogarci/32/24527_2.png) [@edogarci](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fedogarci)%5C
**Post date:** [March 6, 2024, 7:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-tokenizer-finetune-model-or-retrain-model%2F76269%2F1 "2024-03-06T19:51:55Z")

</div>

Hello there!

I’m working on a side project where I’ve been creating a custom dataset (HF Dataset) and a custom tokenizer (HF Tokenizer).

The dataset contains raw text strings representing all moves done in a chess game (each row is a full game). Using this dataset, I trained a custom tokenizer based on Google Bert (AutoTokenizer.from\_pretrained(‘google-bert/bert-base-cased’).

At this point, my plan was to train the model (again Google Bert from pretained, so I used the right tokenizer for its corresponding model), but I have a question…

Should I train the model from the ground, or finetune it? On one hand I think finetune the model should be the right way for most cases (being mine one of them). On the other hand, the vocabulary now available in my custom tokenizer after training it, is way less in size than the original (around 1:10), and therefor (as you can imaging if you are familiar with chess notation), pretty different from english language or any human language.

Any suggestion on what should be the best option? I have choosen google-bert/bert-base-cased because I want to use a fill-mask strategy for training the model.

Best regards,  
Edo

---

<div class="post-metadata">

**Author:** ![Sandy1857](https://avatars.discourse-cdn.com/v4/letter/s/ecae2f/32.png) [@Sandy1857](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSandy1857)%5C
**Post date:** [March 8, 2024, 12:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-tokenizer-finetune-model-or-retrain-model%2F76269%2F2 "2024-03-08T12:08:27Z")

</div>

You might find this discussion helpful:

> **[The Learning Agency Lab - PII Data Detection](https://www.kaggle.com/competitions/pii-detection-removal-from-educational-data/discussion/480557#2674152)**
>
> Develop automated techniques to detect and remove PII from educational data.