# Can we add extra word embedding to the BERT? **URL:** **Author:** ![rohanaiml](https://avatars.discourse-cdn.com/v4/letter/r/ebca7d/32.png) [@rohanaiml](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frohanaiml)%5C **Post date:** [February 21, 2022, 1:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-we-add-extra-word-embedding-to-the-bert%2F14951%2F1 "2022-02-21T01:28:50Z") Hi Everyone, I am trying to add a tfidf-weighted word2vec embedding to the BERT input as an experiment. To generate text using bert\_to\_bert or BERT encoder with other transformer decoder. Thanks, Rohan ---
**Author:** ![beneyal](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/beneyal/32/27988_2.png) [@beneyal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbeneyal)%5C **Post date:** [February 21, 2022, 2:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-we-add-extra-word-embedding-to-the-bert%2F14951%2F2 "2022-02-21T02:19:01Z")
Hello! 😁 I’m not sure I 100% understand what you’re trying to do. If what you want is to add another embedding matrix to the existing word embedding matrix, you can do this: ```python import torch from transformers import BertModel model = BertModel.from_pretrained("bert-base-uncased") with torch.no_grad(): model.embeddings.word_embeddings.add_(word2vec_matrix) ``` If you want another embedding layer, then it’s a bit more complicated. You will need to copy the `BertEmbeddings` layer code from [here](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/modeling_bert.py#L166) and add the new layer there. Once you’ve done that, you can just switch the whole module and re-set the pre-trained weights: ```python with torch.no_grad(): word_embeddings = torch.clone(model.embeddings.word_embeddings.weight) pos_embeddings = torch.clone(model.embeddings.position_embeddings.weight) token_type_embeddings = torch.clone(model.embeddings.token_type_embeddings.weight) config = AutoConfig.from_pretrained("bert-base-uncased") model.embeddings = AugmentedBertEmbeddings(config) with torch.no_grad(): model.embeddings.word_embeddings.weight.set_(word_embeddings) model.embeddings.position_embeddings.weight.set_(pos_embeddings) model.embeddings.token_type_embeddings.weight.set_(token_type_embeddings) ``` I hope I haven’t made any mistakes, and that I managed to help 🙂 ---
**Author:** ![rohanaiml](https://avatars.discourse-cdn.com/v4/letter/r/ebca7d/32.png) [@rohanaiml](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frohanaiml)%5C **Post date:** [February 26, 2022, 3:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-we-add-extra-word-embedding-to-the-bert%2F14951%2F4 "2022-02-26T03:06:00Z")
Thanks @beneyal. This seems complicated as I may need to re-train everything and it is complex. Can we combine BERT embedding and my own embedding before passing it to the decoder? ---
**Author:** ![beneyal](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/beneyal/32/27988_2.png) [@beneyal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbeneyal)%5C **Post date:** [March 2, 2022, 12:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-we-add-extra-word-embedding-to-the-bert%2F14951%2F5 "2022-03-02T12:34:52Z")
BERT doesn’t have a decoder, so I’m not sure what you’re referring to. ---
**Author:** ![hozer](https://avatars.discourse-cdn.com/v4/letter/h/5fc32e/32.png) [@hozer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhozer)%5C **Post date:** [August 4, 2022, 8:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-we-add-extra-word-embedding-to-the-bert%2F14951%2F6 "2022-08-04T20:44:07Z")
Hi @beneyal, I am planning to pre-train BERT with an extra input embedding( For e.g. one input for Wordpiece tokenizer, one for BPE ) Can I make it with the same method you explained? Actually, I am planning to add some syntactic information in the second embedding. Does it make sense? ---
**Author:** ![mobassir](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mobassir/32/5587_2.png) [@mobassir](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmobassir)%5C **Post date:** [August 4, 2022, 10:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-we-add-extra-word-embedding-to-the-bert%2F14951%2F7 "2022-08-04T22:19:08Z")
hello, i think my question here [How to concat laserembeddings with huggingface funnel transformers simple CLS output for fine tuning on downstream NLP sequence classification data problem?](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-concat-laserembeddings-with-huggingface-funnel-transformers-simple-cls-output-for-fine-tuning-on-downstream-nlp-sequence-classification-data-problem%2F21254) is similar? can you please help? thanks in advance. ---
**Author:** ![mobassir](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mobassir/32/5587_2.png) [@mobassir](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmobassir)%5C **Post date:** [August 15, 2022, 6:12pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-we-add-extra-word-embedding-to-the-bert%2F14951%2F8 "2022-08-15T18:12:13Z")
your code doesn’t work. ```auto --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) /tmp/ipykernel_17/1823014925.py in 8 model = BertModel.from_pretrained("bert-base-uncased") 9 with torch.no_grad(): ---> 10 model.embeddings.word_embeddings.add_(fold0_laser) 11 # model.deberta.embeddings.word_embeddings /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in __getattr__ (self, name) 1184 return modules[name] 1185 raise AttributeError("'{}' object has no attribute '{}'".format( -> 1186 type(self). __name__ , name)) 1187 1188 def __setattr__ (self, name: str, value: Union[Tensor, 'Module']) -> None: AttributeError: 'Embedding' object has no attribute 'add_' ```