# Bert Config: Num attention heads

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-config-num-attention-heads%2F15158%3E%5C
**Category:** 🤗Transformers\
**Created:** [February 26, 2022, 11:32am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-config-num-attention-heads%2F15158 "2022-02-26T11:32:12Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ritu](https://avatars.discourse-cdn.com/v4/letter/r/7993a0/32.png) [@ritu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fritu)%5C
**Post date:** [February 26, 2022, 11:32am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-config-num-attention-heads%2F15158%2F1 "2022-02-26T11:32:12Z")

</div>

Hi, I am using the BertConfig to create a encoder decoder model in the following way:

encoder = BertConfig()  
decoder = BertConfig()  
config = EncoderDecoderConfig.from\_encoder\_decoder\_configs(encoder, decoder)  
bert2bert = EncoderDecoderModel(config=config)  
bert2bert.config.decoder.is\_decoder = True  
bert2bert.config.decoder.add\_cross\_attention = True

bert2bert.config.encoder.num\_attention\_heads = 12  
print(bert2bert.encoder.num\_parameters(only\_trainable=True),bert2bert.encoder.config.num\_attention\_heads)

With default attention heads 12 the trainable parameters and attention heads at encoder are as below

> > > 86742528, 12

However when I try to change the number of attention heads to 4, the number of trainable parameters does not change while the value for number of attention heads changes (as below). Can anyone help me out?

* * *

bert2bert.config.decoder.add\_cross\_attention = True  
bert2bert.config.encoder.num\_attention\_heads = 4  
print(bert2bert.encoder.num\_parameters(only\_trainable=True), bert2bert.encoder.config.num\_attention\_heads)

> > > (86742528, 4)

---

<div class="post-metadata">

**Author:** ![ritu](https://avatars.discourse-cdn.com/v4/letter/r/7993a0/32.png) [@ritu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fritu)%5C
**Post date:** [February 27, 2022, 11:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-config-num-attention-heads%2F15158%2F2 "2022-02-27T11:01:09Z")

</div>

@valhalla . Could you may be help me out here ? If this is an expected behavior or not ?

---

<div class="post-metadata">

**Author:** ![shanefeng123](https://avatars.discourse-cdn.com/v4/letter/s/b4bc9f/32.png) [@shanefeng123](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fshanefeng123)%5C
**Post date:** [March 7, 2023, 6:02am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-config-num-attention-heads%2F15158%2F3 "2023-03-07T06:02:32Z")

</div>

The total number of attention parameters won’t change. When there are 12 attention heads, Q, K and V matrices are split into 12 smaller versions as “heads”. When you specify 4 attention heads, the same thing happens.