# Custom BERT Masked LM output embedding

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-bert-masked-lm-output-embedding%2F22656%3E%5C
**Category:** Beginners\
**Created:** [September 7, 2022, 3:20am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-bert-masked-lm-output-embedding%2F22656 "2022-09-07T03:20:49Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![kevintuan](https://avatars.discourse-cdn.com/v4/letter/k/3bc359/32.png) [@kevintuan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkevintuan)%5C
**Post date:** [September 7, 2022, 3:20am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-bert-masked-lm-output-embedding%2F22656%2F1 "2022-09-07T03:20:49Z")

</div>

Hi,  
I trained a custom sense embeddings based on Wordnet definition and tree structure.  
Now I want to test the embeddings by fine tuning BERT masked LM so the model predicts the most likely sense embedding. For example:  
" I need to go to the [bank] today" → bank.wn.02

I’m uncertain how to accomplish this.

1. Can I provide a set of output labels with their embeddings different from the input embedding and input\_ids?
2. Should I build a custom softmax layer from the output\_hidden\_states of the BERTModel