# Visual Tokenization / Masking In BEIT & LayoutLMv3

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fvisual-tokenization-masking-in-beit-layoutlmv3%2F28332%3E%5C
**Category:** Intermediate\
**Created:** [December 23, 2022, 12:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fvisual-tokenization-masking-in-beit-layoutlmv3%2F28332 "2022-12-23T12:51:04Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![mvalente](https://avatars.discourse-cdn.com/v4/letter/m/b9e5f3/32.png) [@mvalente](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmvalente)%5C
**Post date:** [December 23, 2022, 12:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fvisual-tokenization-masking-in-beit-layoutlmv3%2F28332%2F1 "2022-12-23T12:51:05Z")

</div>

**tldr questions** :

1. Why do papers like [LayoutLMv3](https://arxiv.org/pdf/2204.08387.pdf) mention visual tokenizers that are absent from all open-source implementations?
2. Why do most BEIT implementations lack the visual tokenizer?
3. Why do pixel values go through PatchEmbedding straight into a decoder without any tokenization?
4. What is the relationship between Patch Embed, self.mask\_token = nn.Parameter() and visual tokenizers.

**Background** :

I’m coding the heads for [LayoutLMv3](https://arxiv.org/pdf/2204.08387.pdf) pretraining. The paper claims to use the visual tokenizer from [DIT](https://arxiv.org/pdf/2203.02378.pdf) to tokenize image patches. DIT is essentially BEIT with a dVAE trained on documents.

But, when I look at the code in HuggingFace and TIMM for BEIT, there’s no tokenization of image patches.

- [timm: BEIT](https://github.com/rwightman/pytorch-image-models/blob/18ec173f95aa220af753358bf860b16b6691edb2/timm/models/beit.py#L316)
- [hugging-face: BEIT](https://github.com/huggingface/transformers/blob/f7f0ec2f549ca2e0f9190789907919d213179bba/src/transformers/models/beit/modeling_beit.py#L141)

The pixel values get transformed into embeds that go straight into the encoder.

In [Microsofts BEIT implementation](https://github.com/microsoft/unilm/blob/ce11581539796851fba9f3e71744b2119cc2df21/beit/engine_for_pretraining.py#L50) I managed to find an example that makes use of visual tokenizers.

After successfully using the visual tokenizer, my confusion only increased and led me to **question number 4**.

Thank you for reading and for your attention. Any light shed on this subject would be amazing.

---

<div class="post-metadata">

**Author:** ![mvalente](https://avatars.discourse-cdn.com/v4/letter/m/b9e5f3/32.png) [@mvalente](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmvalente)%5C
**Post date:** [December 23, 2022, 1:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fvisual-tokenization-masking-in-beit-layoutlmv3%2F28332%2F2 "2022-12-23T13:03:12Z")

</div>

💡 moment.

Is the `self.mask_token = nn.Parameter(torch.zeros(1, 1, embed_dim))` what we replace the visual embeddings to be masked with? Is the visual token a learnable parameter?