# How to pre-train a model using a custom mask strategy?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-pre-train-a-model-using-a-custom-mask-strategy%2F17447%3E%5C
**Category:** 🤗Transformers\
**Created:** [May 2, 2022, 8:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-pre-train-a-model-using-a-custom-mask-strategy%2F17447 "2022-05-02T20:44:26Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![vabatista](https://avatars.discourse-cdn.com/v4/letter/v/47e85d/32.png) [@vabatista](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvabatista)%5C
**Post date:** [May 2, 2022, 8:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-pre-train-a-model-using-a-custom-mask-strategy%2F17447%2F1 "2022-05-02T20:44:26Z")

</div>

Is there any way to pre-train a model (i.e. RoBERTa), using my own corpus, but with a custom mask strategy? For instance, I want to look up a dictionary to choose which words to mask.