# Strange output using BioBERT for imputing MASK tokens
**URL:**
**Author:**  [@rahuln](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frahuln)%5C
**Post date:** [December 31, 2020, 10:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstrange-output-using-biobert-for-imputing-mask-tokens%2F3014%2F1 "2020-12-31T22:13:08Z")
I’m trying to use BioBERT (downloaded from the HuggingFace models repository at `dmis-lab/biobert-v1.1`) to fill in MASK tokens in text, and I’m getting some unexpected behavior with the suggested tokens.
I pasted a screenshot below comparing `bert-base-uncased` (which behaves as expected and has sensible most-likely tokens) with BioBERT:
Here’s the code to reproduce this:
```auto
from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch
text = 'heart disease is [MASK] leading cause of death in the united states.'
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
model = AutoModelForMaskedLM.from_pretrained('bert-base-uncased')
tokenized = tokenizer(text, return_tensors='pt')
idx = tokenizer.convert_ids_to_tokens(tokenized.input_ids[0]).index(tokenizer.mask_token)
output = model(**tokenized, return_dict=True)
print(tokenizer.convert_ids_to_tokens(torch.topk(output.logits[0, idx, :], 10).indices))
tokenizer = AutoTokenizer.from_pretrained('dmis-lab/biobert-v1.1')
model = AutoModelForMaskedLM.from_pretrained('dmis-lab/biobert-v1.1')
tokenized = tokenizer(text, return_tensors='pt')
idx = tokenizer.convert_ids_to_tokens(tokenized.input_ids[0]).index(tokenizer.mask_token)
output = model(**tokenized, return_dict=True)
print(tokenizer.convert_ids_to_tokens(torch.topk(output.logits[0, idx, :], 10).indices))
```
And here’s my output from running `transformers-cli env`:
```auto
- `transformers` version: 4.1.1
- Platform: macOS-10.11.6-x86_64-i386-64bit
- Python version: 3.8.5
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
```
I also asked about similar issues with PubMedBERT as a [Github issue](https://github.com/huggingface/transformers/issues/8505) a while back, but haven’t gotten a response.
Do the pretrained weights for these models not contain the components necessary for doing masked language modeling/imputing MASK tokens? Is there any way to fix this issue?
---
**Author:**  [@rgwatwormhill](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frgwatwormhill)%5C
**Post date:** [January 4, 2021, 5:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstrange-output-using-biobert-for-imputing-mask-tokens%2F3014%2F2 "2021-01-04T17:18:25Z")
Hi,
I am not an expert, but that is what it looks like to me.
Masked Language Modelling is usually used during pre-training, and is often not needed during fine-tuning, so I guess the DIMS team didn’t think the MLM parameters would be required.
I notice that the DIMS team have provided 5 models. Do any of the other models have MLM parameters?
It should certainly be possible to copy the DIMS weights into your own model, where your own model does include an MLM head. I expect you would then need to train your model before it would give sensible answers, unless you could find a suitable MLM head to copy (probably not…).
---
**Author:**  [@BramVanroy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBramVanroy)%5C
**Post date:** [January 4, 2021, 6:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstrange-output-using-biobert-for-imputing-mask-tokens%2F3014%2F3 "2021-01-04T18:28:23Z")
The reason you are not getting a response is because this is near impossible to debug: these are third-party models that someone else trained. It is possible that they did not train/finetune these models on MLM and in such event the model doesn’t know what to output for the mask.
You should try to get into contact with the model creators to get an answer.