# Costumizing MASKed tokens

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcostumizing-masked-tokens%2F51257%3E%5C
**Category:** 🤗Transformers\
**Created:** [August 18, 2023, 3:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcostumizing-masked-tokens%2F51257 "2023-08-18T15:29:22Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![svallejovera](https://avatars.discourse-cdn.com/v4/letter/s/9dc877/32.png) [@svallejovera](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsvallejovera)%5C
**Post date:** [August 18, 2023, 3:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcostumizing-masked-tokens%2F51257%2F1 "2023-08-18T15:29:22Z")

</div>

Hello,

I want to further pre-train a RoBERTa model using a customized set of masked tokens (e.g., masking a percentage of high information words or never masking stopwords). Is there a way to do this?

This post asked a similar question, but there is no answer:

> [@How to pre-train a model using a custom mask strategy?](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-pre-train-a-model-using-a-custom-mask-strategy%2F17447)%3A
>
> Is there any way to pre-train a model (i.e. RoBERTa), using my own corpus, but with a custom mask strategy? For instance, I want to look up a dictionary to choose which words to mask.

---

<div class="post-metadata">

**Author:** ![bk073](https://avatars.discourse-cdn.com/v4/letter/b/8491ac/32.png) [@bk073](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbk073)%5C
**Post date:** [September 27, 2023, 6:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcostumizing-masked-tokens%2F51257%2F2 "2023-09-27T18:23:52Z")

</div>

Were you able to solve it?