# BertSelfAttention, BertSelfOutput implementation

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertselfattention-bertselfoutput-implementation%2F21439%3E%5C
**Category:** 🤗Transformers\
**Created:** [August 9, 2022, 7:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertselfattention-bertselfoutput-implementation%2F21439 "2022-08-09T19:49:19Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![heekang](https://avatars.discourse-cdn.com/v4/letter/h/51bf81/32.png) [@heekang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fheekang)%5C
**Post date:** [August 9, 2022, 7:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertselfattention-bertselfoutput-implementation%2F21439%2F1 "2022-08-09T19:49:19Z")

</div>

Hi, I’m studying BERT and am curious about the implementation of `BertSelfAttention` and `BertSelfOutput`.

According to the Transformer paper([Attention Is All You Need (2017, Vaswani et el.)](https://arxiv.org/abs/1706.03762)), multi-head self attention requires 4 Linear layers: one for query, one for key, one for value, and the last for final output.

However, `BertSelfAttention` only has three Linear layers; only for query, key, and value. The Linear layer for the final output is in `BertSelfOutput`.

Is there any reason why they are implemented like this?

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [August 10, 2022, 9:51am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertselfattention-bertselfoutput-implementation%2F21439%2F2 "2022-08-10T09:51:41Z")

</div>

I think this was a design decision by @thomwolf.

---

<div class="post-metadata">

**Author:** ![thomwolf](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thomwolf/32/2198_2.png) [@thomwolf](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fthomwolf)%5C
**Post date:** [August 10, 2022, 12:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertselfattention-bertselfoutput-implementation%2F21439%2F3 "2022-08-10T12:33:17Z")

</div>

It was done like this in the original tensorflow implementation of Bert and can probably be traced back to the tensor2tensor library

---

<div class="post-metadata">

**Author:** ![ccdv](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ccdv/32/28048_2.png) [@ccdv](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fccdv)%5C
**Post date:** [August 10, 2022, 4:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertselfattention-bertselfoutput-implementation%2F21439%2F4 "2022-08-10T16:33:03Z")

</div>

Its model/original implementation dependent, DistilBERT does the 4th linear transformation for example.

---

<div class="post-metadata">

**Author:** ![heekang](https://avatars.discourse-cdn.com/v4/letter/h/51bf81/32.png) [@heekang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fheekang)%5C
**Post date:** [August 11, 2022, 7:31am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertselfattention-bertselfoutput-implementation%2F21439%2F5 "2022-08-11T07:31:58Z")

</div>

Thank you all for your reply:-)