# \[unused\] tokens in predicting with MLM model

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funused-tokens-in-predicting-with-mlm-model%2F13344%3E%5C
**Category:** Beginners\
**Created:** [January 3, 2022, 8:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funused-tokens-in-predicting-with-mlm-model%2F13344 "2022-01-03T20:59:55Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![MahdiA](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mahdia/32/2700_2.png) [@MahdiA](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMahdiA)%5C
**Post date:** [January 3, 2022, 8:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funused-tokens-in-predicting-with-mlm-model%2F13344%2F1 "2022-01-03T20:59:56Z")

</div>

I did domain adaptation over my dataset using BERT using MLM model.  
When I wanted to test the model with some `[MASK]` tokens, it returned some `[unused177]` tokens as top\_k. So, where this problem come from? Is that related to tokenizer?

```python
from transformers import DistilBertTokenizer
from torch.nn import functional as F

tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
model = trainer.model.base_model # from Trainer class

text = "I am going to get " + tokenizer.mask_token + " vaccine"
input1 = tokenizer.encode_plus(text, return_tensors = "pt")

mask_index = torch.where(input1["input_ids"][0] == tokenizer.mask_token_id)

logits = model(**input1)
logits = logits.last_hidden_state

softmax = F.softmax(logits, dim = -1)
print(softmax.shape)
print(mask_index)
mask_word = softmax[0, mask_index, :]
top_word = torch.argmax(mask_word, dim=1)
print(tokenizer.decode(top_word))

```