# Metrics for masked language modeling (mlm)

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmetrics-for-masked-language-modeling-mlm%2F10055%3E%5C
**Category:** Beginners\
**Created:** [September 16, 2021, 8:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmetrics-for-masked-language-modeling-mlm%2F10055 "2021-09-16T20:17:44Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![MilanKalkenings](https://avatars.discourse-cdn.com/v4/letter/m/a183cd/32.png) [@MilanKalkenings](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMilanKalkenings)%5C
**Post date:** [September 16, 2021, 8:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmetrics-for-masked-language-modeling-mlm%2F10055%2F1 "2021-09-16T20:17:45Z")

</div>

i want to compare the fit of my bert model before and after performing mlm for some epochs on my own textual data.

according to [Perplexity of fixed-length models — transformers 4.10.1 documentation](https://huggingface.co/transformers/perplexity.html) perplexity isn’t well defined for mlm. which metric should i use instead? so far i calculated the accuracy on the masked tokens only (compare actual labels of masked inputs with the token predicted by the model for the masked position. this is obviously not a good metric, since it punishes synonyms etc. like any other false prediction)