# Pre-trained DeBERTa - Weak MLM performance any hints?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-trained-deberta-weak-mlm-performance-any-hints%2F43878%3E%5C
**Category:** Research\
**Created:** [June 20, 2023, 7:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-trained-deberta-weak-mlm-performance-any-hints%2F43878 "2023-06-20T07:34:23Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mintbach](https://avatars.discourse-cdn.com/v4/letter/m/35a633/32.png) [@Mintbach](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMintbach)%5C
**Post date:** [June 20, 2023, 7:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-trained-deberta-weak-mlm-performance-any-hints%2F43878%2F1 "2023-06-20T07:34:23Z")

</div>

Hi,

I wanted to use DeBERTa.  
Somehow the preview of its unmasking abilities seems very bad.

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/b/2/b287438c935d61369dc9aa5f43a84d7ba358269f.png)

I looked at the source code and cannot see the addition of the absolute positions.

Can someone explain me why the model performs so bad at MLM preview.  
Maybe I overlooked the addition of the absolute positions in the source code.  
An explanation of the implementation would be really helpful aswell!

Thank you!  
Stephan

---

<div class="post-metadata">

**Author:** ![tgsc](https://avatars.discourse-cdn.com/v4/letter/t/8baadc/32.png) [@tgsc](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftgsc)%5C
**Post date:** [July 21, 2023, 12:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-trained-deberta-weak-mlm-performance-any-hints%2F43878%2F2 "2023-07-21T00:28:05Z")

</div>

Is this deberta v3? The thing is that debertav3 is the discriminator trained with Replaced Token Detection, not MLM. Altought at somepoint they’ve added the MLM heads, in their work they didn’t mention anything like running tests on the discriminators with MLM tasks.

Basically, MLM should yield really bad result with the discriminator, like it is. You should download the generator model (the file pytoroch\_model.generator.bin and generator\_config.json on xsmall, large or mdeberta, it’s missing on base model) and MLM will run just fine.