# LayoutLMV3 embeddings
**URL:**
**Author:**  [@Kforcode](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKforcode)%5C
**Post date:** [August 3, 2022, 6:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flayoutlmv3-embeddings%2F21187%2F1 "2022-08-03T06:54:36Z")
hi @neilsr @sgugger
I saw layoutlmv3 output embedding is (709, 768). which is greater than the **max\_position\_embeddings** = 512.
So i was wondering if the rest (709-512) = 197 is for image embeddings?
I read the paper it says
```auto
, the last layer outputs text-and-image
contextual representations
```
So is there a way we can seprate image and text embedding , like we can in layoutlmv2
if you can point me to any resource explaining this it would be great.
thanks
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [August 3, 2022, 1:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flayoutlmv3-embeddings%2F21187%2F2 "2022-08-03T13:05:47Z")
Hi,
yes the final embeddings (hidden states) that come out of LayoutLMv3Model are for both the text and image tokens. The final hidden states (`outputs.last_hidden_state`) is a tensor of shape (batch\_size, seq\_len, hidden\_size), where the sequence length equals the number of text tokens + the number of image tokens + 1 (we add 1 for the special CLS token, which is useful for classification).
You can separate them by checking the length of the input\_ids you sent through the model, like so:
```auto
text_seq_length = input_ids.shape[1]
# only take the text part of the output representations
sequence_output = outputs.last_hidden_state[:, :text_seq_length]
```
---
**Author:**  [@Kforcode](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKforcode)%5C
**Post date:** [August 3, 2022, 1:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flayoutlmv3-embeddings%2F21187%2F3 "2022-08-03T13:28:43Z")
thanks a lot, @nielsr following the same
is the [cls] token is at last here?
the last token should be [sep] right. so it should be
[cls] [512 \* text tokens] [sep] [num\_patches \* image tokens] . right?
can you please clarify more on the sequence of these tokens
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [August 3, 2022, 1:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flayoutlmv3-embeddings%2F21187%2F4 "2022-08-03T13:57:51Z")
No there’s no SEP token used, it’s just:
```auto
[CLS], text tokens, image tokens
```
as can be seen [here](https://github.com/huggingface/transformers/blob/8fb7c908c81de73bc8b525cdbbb4035dcc39581f/src/transformers/models/layoutlmv3/modeling_layoutlmv3.py#L875) - where the text embeddings and visual embeddings are concatenated.
The only time you’re using a [SEP] token is when doing question answering, then you’ll have
```auto
[CLS], question tokens, [SEP], context tokens, image tokens
```
---
**Author:**  [@Kforcode](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKforcode)%5C
**Post date:** [August 3, 2022, 2:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flayoutlmv3-embeddings%2F21187%2F5 "2022-08-03T14:05:56Z")
cool thanks