# Evaluation metrics for BERT-like LMs
**URL:**
**Author:**  [@vblagoje](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvblagoje)%5C
**Post date:** [September 24, 2020, 8:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fevaluation-metrics-for-bert-like-lms%2F1256%2F1 "2020-09-24T20:16:34Z")
Hey guys,
I’ve read that Perplexity (PPL) is one of the most common metrics for evaluating autoregressive and causal language models. But what do we use for MLMs like BERT?
I need to evaluate BERT models after pre-training and compare them to existing BERT models without going through downstream task GLUE-like benchmarks.
Best,
Vladimir
---
**Author:**  [@vblagoje](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvblagoje)%5C
**Post date:** [September 25, 2020, 12:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fevaluation-metrics-for-bert-like-lms%2F1256%2F2 "2020-09-25T12:33:24Z")
I found an interesting project [https://github.com/awslabs/mlm-scoring](https://github.com/awslabs/mlm-scoring) which seems to be the step in the right direction. The authors also published the paper [https://arxiv.org/pdf/1910.14659v2.pdf](https://arxiv.org/pdf/1910.14659v2.pdf)
---
**Author:**  [@stefan-it](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fstefan-it)%5C
**Post date:** [October 4, 2020, 12:23am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fevaluation-metrics-for-bert-like-lms%2F1256%2F3 "2020-10-04T00:23:22Z")
Hi Vladimir,
before releasing new models, I usually perform evaluations for multiple checkpoints on at least two downstream tasks (normally Pos tagging or NER).
But maybe you can also evaluate the MLM capability for some checkpoints, like it is shown in the following paper:
> **[GitHub - TurkuNLP/bert-eval](https://github.com/TurkuNLP/bert-eval)**
>
> Contribute to TurkuNLP/bert-eval development by creating an account on GitHub.
I would use the “Cloze test word prediction” task. It masks out some subwords from an input sentence, tries to re-construct the masked subwords and calculates accuracy. With that task you could at least measure the MLM capability of your checkpoints, without performing extensive hyper-parameter search and multiple runs as you do for down-stream tasks.
---
**Author:**  [@vblagoje](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvblagoje)%5C
**Post date:** [October 5, 2020, 2:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fevaluation-metrics-for-bert-like-lms%2F1256%2F4 "2020-10-05T14:58:52Z")
Thanks a lot @stefan-it I see the project is using the old HF naming scheme but it shouldn’t be hard to update.
---
**Author:**  [@sxdmit](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsxdmit)%5C
**Post date:** [December 6, 2024, 12:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fevaluation-metrics-for-bert-like-lms%2F1256%2F5 "2024-12-06T12:03:06Z")
make it a bit simple, you can always trust chatgpt. he knows it.