# Mask More Than one Word:

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmask-more-than-one-word%2F3800%3E%5C
**Category:** 🤗Transformers\
**Created:** [February 22, 2021, 4:22am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmask-more-than-one-word%2F3800 "2021-02-22T04:22:14Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![zanderbush](https://avatars.discourse-cdn.com/v4/letter/z/13edae/32.png) [@zanderbush](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzanderbush)%5C
**Post date:** [February 22, 2021, 4:22am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmask-more-than-one-word%2F3800%2F1 "2021-02-22T04:22:14Z")

</div>

![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/9/9a997ca44568d1a221ebf26e082ccc16956e38d2.png)

Here, it says you can mask k tokens. However, in the documentation, it shows you only being able to mask one token. Is it possible to mask k words or am I mistaken?

---

<div class="post-metadata">

**Author:** ![blorg](https://avatars.discourse-cdn.com/v4/letter/b/bbe5ce/32.png) [@blorg](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fblorg)%5C
**Post date:** [February 22, 2021, 11:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmask-more-than-one-word%2F3800%2F2 "2021-02-22T23:42:14Z")

</div>

Are you using a fill-mask pipeline? If so, there’s a hard-coded limit of a single mask in the class, even though the model itself may support multiple masks. I guess the added behavior would warrant some more functionality when it comes to choosing how to sample, e.g. if there are N masked tokens with a few top-k probabilities each one might either want to sample from the join distribution (i.e. ranking the pairs based on p1\*p2) or independently. The best approach would depend on the model’s internals, I suppose.

I saw a post a while back welcoming a PR for this matter, so it’s a wanted feature.

---

<div class="post-metadata">

**Author:** ![zanderbush](https://avatars.discourse-cdn.com/v4/letter/z/13edae/32.png) [@zanderbush](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzanderbush)%5C
**Post date:** [February 23, 2021, 5:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmask-more-than-one-word%2F3800%2F3 "2021-02-23T17:08:02Z")

</div>

I was using the following. However, this code does not work well if you’re aiming to consecutive tokens.

```
import torch

sentence = "The capital of France <mask> contains the Eiffel <mask>."

token_ids = tokenizer.encode(sentence, return_tensors='pt')

# print(token_ids)

token_ids_tk = tokenizer.tokenize(sentence, return_tensors='pt')

print(token_ids_tk)

masked_position = (token_ids.squeeze() == tokenizer.mask_token_id).nonzero()

masked_pos = [mask.item() for mask in masked_position]

print (masked_pos)

with torch.no_grad():

    output = model(token_ids)

last_hidden_state = output[0].squeeze()

print ("\n\n")

print ("sentence : ",sentence)

print ("\n")

list_of_list =[]

for mask_index in masked_pos:

    mask_hidden_state = last_hidden_state[mask_index]

    idx = torch.topk(mask_hidden_state, k=100, dim=0)[1]

    words = [tokenizer.decode(i.item()).strip() for i in idx]

    list_of_list.append(words)

    print (words)

best_guess = ""

for j in list_of_list:

    best_guess = best_guess+" "+j[0]

```

`---`

```
What I may try to do is <mask> two consecutive tokens. 

(ex: Paris is <mask> <mask> to visit.)

I'll then re-insert the most probable token for the first <mask>. 

(ex: Paris is a <mask> to visit.)

Then, I'll return the second <mask>'s most probable token. 

(ex: Paris is a city to visit.)
```

---

<div class="post-metadata">

**Author:** ![zanderbush](https://avatars.discourse-cdn.com/v4/letter/z/13edae/32.png) [@zanderbush](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzanderbush)%5C
**Post date:** [February 23, 2021, 5:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmask-more-than-one-word%2F3800%2F4 "2021-02-23T17:34:19Z")

</div>

This is the updated code.

```
import torch

sentence = "The capital of France <mask> <mask> the Eiffel Tower."

token_ids = tokenizer.encode(sentence, return_tensors='pt')

token_ids_tk = tokenizer.tokenize(sentence, return_tensors='pt')

print(token_ids_tk)

masked_position = (token_ids.squeeze() == tokenizer.mask_token_id).nonzero()

masked_pos = [mask.item() for mask in masked_position]

print (masked_pos)

with torch.no_grad():

    output = model(token_ids)

last_hidden_state = output[0].squeeze()

print ("\n\n")

print ("sentence : ",sentence)

print ("\n")

list_of_list =[]

for mask_index in masked_pos:

    mask_hidden_state = last_hidden_state[mask_index]

    idx = torch.topk(mask_hidden_state, k=5, dim=0)[1]

    words = [tokenizer.decode(i.item()).strip() for i in idx]

    list_of_list.append(words)

    #print(words)

sentences3 = []

for i in range(5):

    New = (list_of_list[0])

    New = New[i]

    New = New.replace("['", "").replace("']", "")

    sentence2 = sentence.replace("<mask>", New, 1)

    sentences3.append(sentence2)

for i in sentences3:

    #print(i)

    token_ids = tokenizer.encode(i, return_tensors='pt')

    token_ids_tk = tokenizer.tokenize(i, return_tensors='pt')

    #print(token_ids_tk)

    masked_position = (token_ids.squeeze() == tokenizer.mask_token_id).nonzero()

    masked_pos = [mask.item() for mask in masked_position]

    print (masked_pos)

    with torch.no_grad():

        output = model(token_ids)

        last_hidden_state = output[0].squeeze()

        print ("\n\n")

        print ("sentence : ",i)

        print ("\n")

        list_of_list =[]

        for mask_index in masked_pos:

            mask_hidden_state = last_hidden_state[mask_index]

            idx = torch.topk(mask_hidden_state, k=5, dim=0)[1]

            words = [tokenizer.decode(i.item()).strip() for i in idx]

            list_of_list.append(words)

            print(words)

```

`-`

```
sentence : The capital of France <mask> <mask> the Eiffel Tower.

[6]

sentence : The capital of France , <mask> the Eiffel Tower.

['and', 'with', 'near', 'at', 'including']
[6]

sentence : The capital of France is <mask> the Eiffel Tower.

['in', 'now', 'called', 'at', 'under']
[6]

sentence : The capital of France lies <mask> the Eiffel Tower.

['atop', 'under', 'beneath', 'in', 'behind']
[6]

sentence : The capital of France stands <mask> the Eiffel Tower.

['atop', 'at', 'on', 'under', 'in']
[6]

sentence : The capital of France rests <mask> the Eiffel Tower.

['atop', 'on', 'in', 'upon', 'under']
```

---

<div class="post-metadata">

**Author:** ![franfram](https://avatars.discourse-cdn.com/v4/letter/f/c6cbf5/32.png) [@franfram](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffranfram)%5C
**Post date:** [October 17, 2022, 1:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmask-more-than-one-word%2F3800%2F5 "2022-10-17T13:48:44Z")

</div>

Hey, thanks for the code.  
Did you manage to solve it for consecutive masked tokens? I’m having the same problem

Cheers,  
Fran

---

<div class="post-metadata">

**Author:** ![cnut1648](https://avatars.discourse-cdn.com/v4/letter/c/77aa72/32.png) [@cnut1648](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcnut1648)%5C
**Post date:** [October 20, 2022, 6:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmask-more-than-one-word%2F3800%2F6 "2022-10-20T18:21:47Z")

</div>

@franfram I had the same issue… Do you happen to know which model supports multiple tokens during inference?

---

<div class="post-metadata">

**Author:** ![franfram](https://avatars.discourse-cdn.com/v4/letter/f/c6cbf5/32.png) [@franfram](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffranfram)%5C
**Post date:** [October 21, 2022, 2:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmask-more-than-one-word%2F3800%2F7 "2022-10-21T14:33:11Z")

</div>

@cnut1648  
I did, but it doesn’t work that well (I’m using a Spanish model, maybe an English one will work better).  
here’s a collab with the code

hope it helps  
cheers

> **[GitHub - franfram/Multi-Mask-filling-model: Multi-mask filling model](https://github.com/franfram/Multi-Mask-filling-model)**
>
> Multi-mask filling model . Contribute to franfram/Multi-Mask-filling-model development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![cnut1648](https://avatars.discourse-cdn.com/v4/letter/c/77aa72/32.png) [@cnut1648](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcnut1648)%5C
**Post date:** [October 24, 2022, 4:41am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmask-more-than-one-word%2F3800%2F8 "2022-10-24T04:41:03Z")

</div>

I see, thanks!!! I tried with English one and it seems that roberta for English also doesn’t work well…