# How to get a model's initial input representation?
**URL:**
**Author:**  [@yabramuvdi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fyabramuvdi)%5C
**Post date:** [June 16, 2022, 10:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-get-a-models-initial-input-representation%2F19239%2F1 "2022-06-16T10:06:54Z")
Hi! I am trying to recover the initial input representation that BERT generates for each token in a given sequence. I know this initial representation is made by adding the token embedding with the positional and the segment embedding (see picture below from the paper). I also know these individual embeddings (token, positional and segment) can be accessed through the model. However, is there anyway I can extract the complete initial input representation from the output of the model?
I am currently just doing this and have not been able to find this representation anywhere in the output.
```python
# load the model
model = BertModel.from_pretrained('bert-base-uncased',
output_hidden_states = True,
output_attentions = True
)
# define the sentence and tokenize it
sent = "my dog is cute"
encoded_sent = tokenizer(sent, return_tensors="pt")
# pass the sentence through the model
output = model(**encoded_sent)
```
Thanks!
---
**Author:**  [@dblakely](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdblakely)%5C
**Post date:** [June 17, 2022, 8:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-get-a-models-initial-input-representation%2F19239%2F2 "2022-06-17T20:46:08Z")
It doesn’t output the embeddings, but you could make a custom subclass of `BertModel` that does. For example, you could override [this](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/modeling_bert.py#L916) method and subclass the `BaseModelOutputWithPoolingAndCrossAttentions` class to extract the embeddings.
---
**Author:**  [@yabramuvdi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fyabramuvdi)%5C
**Post date:** [June 21, 2022, 8:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-get-a-models-initial-input-representation%2F19239%2F3 "2022-06-21T08:24:08Z")
Hi Derrick! Great idea. Thanks a lot for the recommendation. I will implement this.