# Masking Probability

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmasking-probability%2F746%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [August 18, 2020, 9:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmasking-probability%2F746 "2020-08-18T09:19:27Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![abdallah197](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/abdallah197/32/3973_2.png) [@abdallah197](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabdallah197)%5C
**Post date:** [August 18, 2020, 9:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmasking-probability%2F746%2F1 "2020-08-18T09:19:28Z")

</div>

Hi

I am wondering whether the masking of tokens [MASK] is done by applying the masking probability to a given sequence or the whole batch altogether.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [August 18, 2020, 12:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmasking-probability%2F746%2F2 "2020-08-18T12:21:43Z")

</div>

In [`DataCollatorForLanguageModeling`](https://github.com/huggingface/transformers/blob/master/src/transformers/data/data_collator.py#L120) the masking is done on the tensor directly.

---

<div class="post-metadata">

**Author:** ![RichardWang](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/richardwang/32/45198_2.png) [@RichardWang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRichardWang)%5C
**Post date:** [August 18, 2020, 12:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmasking-probability%2F746%2F3 "2020-08-18T12:49:21Z")

</div>

If I read the code correctly, it is every “token” has masking probability to be masked (makes/replaced/not changed). Independent to how many tokens you have i.e. the size of tensor.

---

<div class="post-metadata">

**Author:** ![abdallah197](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/abdallah197/32/3973_2.png) [@abdallah197](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabdallah197)%5C
**Post date:** [August 19, 2020, 2:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmasking-probability%2F746%2F4 "2020-08-19T14:44:12Z")

</div>

In the paper, It’s mentioned that the masing is done on 0.15 f all WordPiece tokens in each sequence at random.  
In the code though, it’s based on the **inputs** in

```auto
labels = inputs.clone()
probability_matrix = torch.full(labels.shape, self.mlm_probability)
masked_indices = torch.bernoulli(probability_matrix).bool()

```

I don’t know whether inputs refers to one sequence or one batch of sequences

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [August 20, 2020, 3:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmasking-probability%2F746%2F5 "2020-08-20T15:28:31Z")

</div>

`inputs` is batch and `probability_matrix` creates prob for each sequence.