# Bert embedding layer

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-embedding-layer%2F13355%3E%5C
**Category:** Beginners\
**Created:** [January 4, 2022, 8:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-embedding-layer%2F13355 "2022-01-04T08:46:49Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ShivaniSri](https://avatars.discourse-cdn.com/v4/letter/s/3ab097/32.png) [@ShivaniSri](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FShivaniSri)%5C
**Post date:** [January 4, 2022, 8:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-embedding-layer%2F13355%2F1 "2022-01-04T08:46:49Z")

</div>

I have taken specific word embeddings and considered bert model with those embeddings

self.bert = BertModel.from\_pretrained(‘bert-base-uncased’)  
self.bert(inputs\_embeds=x,attention\_mask=attention\_mask, \*args, \*\*kwargs)

Does this means I’m replacing the bert input embeddings(token+position+segment embeddings)

How to consider all embeddings i.e.,(token+position+segment+custom embeddings)

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [January 4, 2022, 4:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-embedding-layer%2F13355%2F2 "2022-01-04T16:54:19Z")

</div>

Hi,

As you can see [here](https://github.com/huggingface/transformers/blob/19d37c2dd36d73537a6855c56808e524fb584459/src/transformers/models/bert/modeling_bert.py#L213), if you provide `inputs_embeds` yourself, they will only be used to replace the token embeddings. The token type and position embeddings will be added separately.