# Sizes of Query, key and value vector in Bert Model
**URL:**
**Author:**  [@Khondoker](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKhondoker)%5C
**Post date:** [September 13, 2020, 4:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsizes-of-query-key-and-value-vector-in-bert-model%2F1102%2F1 "2020-09-13T16:10:13Z")
I have a question about the sizes of query, key and value vectors. As mentioned in this [paper](https://arxiv.org/pdf/1906.04341.pdf) and also demonstrated in this [medium](https://towardsdatascience.com/deconstructing-bert-part-2-visualizing-the-inner-workings-of-attention-60a16d86b5c1), we should be expecting the sizes of query, key and value vectors as [seq\_length x seq\_length]. But when I print the sizes of the parameter like below, I see the sizes of those vectors as [768 x 768].
```
for name, param in model.named_parameters():
print(name, param.size())
>>> bert.bert.encoder.layer.0.attention.self.query.weight torch.Size([768, 768])
bert.bert.encoder.layer.0.attention.self.key.weight torch.Size([768, 768])
bert.bert.encoder.layer.0.attention.self.value.weight torch.Size([768, 768])
```
I am really confused. I feel like I am missing something, could someone please help me figure it out?
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [September 14, 2020, 1:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsizes-of-query-key-and-value-vector-in-bert-model%2F1102%2F2 "2020-09-14T13:41:34Z")
You are looking at the weights of the query/key/value heads, not the value vectors.
---
**Author:**  [@Khondoker](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKhondoker)%5C
**Post date:** [September 14, 2020, 3:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsizes-of-query-key-and-value-vector-in-bert-model%2F1102%2F3 "2020-09-14T15:42:07Z")
Oh I see. Could you please mind explaining how vectors are derived from this weights? or give any references for it?
Thank you
---
**Author:**  [@simonschoe](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsimonschoe)%5C
**Post date:** [March 25, 2021, 9:43am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsizes-of-query-key-and-value-vector-in-bert-model%2F1102%2F4 "2021-03-25T09:43:29Z")
If you dive into the source code of `class BertSelfAttention(nn.Module)` you will find:
```
self.num_attention_heads = config.num_attention_heads
self.attention_head_size = int(config.hidden_size / config.num_attention_heads)
self.all_head_size = self.num_attention_heads * self.attention_head_size
self.query = nn.Linear(config.hidden_size, self.all_head_size)
self.key = nn.Linear(config.hidden_size, self.all_head_size)
self.value = nn.Linear(config.hidden_size, self.all_head_size)
```
In my view, this illustrates that the size of Q, K and V in a single head is actually `768 x 64` if the number of heads is 12. Due to the multi-head attention these are later concatenated to again produce the abovementioned shape of `768 x 768` which you have obeserved.