# Causal masks in BERT vs. GPT2

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcausal-masks-in-bert-vs-gpt2%2F1540%3E%5C
**Category:** Intermediate\
**Created:** [October 14, 2020, 9:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcausal-masks-in-bert-vs-gpt2%2F1540 "2020-10-14T21:43:41Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![dilawn](https://avatars.discourse-cdn.com/v4/letter/d/d2c977/32.png) [@dilawn](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdilawn)%5C
**Post date:** [October 14, 2020, 9:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcausal-masks-in-bert-vs-gpt2%2F1540%2F1 "2020-10-14T21:43:41Z")

</div>

Hi all - just a quick clarification about causal masks in GPT2 and BertLMHeadModel. Its clear that GPT2 automatically adds causal masks (lines 115 and 151 in modeling\_gpt2.py). I believe that this should be the case for BertLMHeadModel as well - my understanding is that it is mainly meant to be used as a decoder in the EncoderDecoder construct (please correct me if this is not the case). However, I am having a hard time finding where this occurs in the source. Are causal masks automatically added? I suspect I am just missing where this happens in the code as the attention\_mask input (meant for masking padding tokens) wouldn’t take care of this. Thanks!

---

<div class="post-metadata">

**Author:** ![claartje-barkhof](https://avatars.discourse-cdn.com/v4/letter/c/71c47a/32.png) [@claartje-barkhof](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fclaartje-barkhof)%5C
**Post date:** [October 26, 2020, 4:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcausal-masks-in-bert-vs-gpt2%2F1540%2F2 "2020-10-26T16:13:25Z")

</div>

Hi @dilawn, have you figured this out yet by any chance? I have just posted a similar question on not finding causal masks in RobertaForCausalLM and BertGeneration classes. I am just wondering if I misunderstand the concept or missing the line of code that applies these causal masks.

[Link to my post](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fforward-looking-or-left-context-attention-mask-left-to-right-generation-with-bertgeneration-and-robertaforcausallm%2F1753%2F3).

---

<div class="post-metadata">

**Author:** ![claartje-barkhof](https://avatars.discourse-cdn.com/v4/letter/c/71c47a/32.png) [@claartje-barkhof](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fclaartje-barkhof)%5C
**Post date:** [October 27, 2020, 6:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcausal-masks-in-bert-vs-gpt2%2F1540%2F3 "2020-10-27T06:50:05Z")

</div>

For me the answer was in the function `get_extended_attention_mask` in [modeling utils](https://huggingface.co/transformers/_modules/transformers/modeling_utils.html).

---

<div class="post-metadata">

**Author:** ![Jung](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jung/32/21989_2.png) [@Jung](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJung)%5C
**Post date:** [October 27, 2020, 7:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcausal-masks-in-bert-vs-gpt2%2F1540%2F4 "2020-10-27T07:29:38Z")

</div>

Yes, I believe that @claartje-barkhof is correct as in `get_extended_attention_mask` there is a check on `config.is_decoder` and the causual mask creation (see line 236 - 255)

---

<div class="post-metadata">

**Author:** ![edmond](https://avatars.discourse-cdn.com/v4/letter/e/fbc32d/32.png) [@edmond](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fedmond)%5C
**Post date:** [December 30, 2022, 10:36am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcausal-masks-in-bert-vs-gpt2%2F1540%2F5 "2022-12-30T10:36:55Z")

</div>

I think only GPT2 is autoregressive, hence the need for causal masks (I guess everywhere) in the architecture.