# Doubts about attention masks

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoubts-about-attention-masks%2F29356%3E%5C
**Category:** 🤗Transformers\
**Created:** [January 11, 2023, 10:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoubts-about-attention-masks%2F29356 "2023-01-11T10:46:58Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![AfonsoSousa](https://avatars.discourse-cdn.com/v4/letter/a/5f9b8f/32.png) [@AfonsoSousa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAfonsoSousa)%5C
**Post date:** [January 11, 2023, 10:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoubts-about-attention-masks%2F29356%2F1 "2023-01-11T10:46:58Z")

</div>

Hi. My issue relates to [this issue](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpass-a-custom-mask-when-using-roberta%2F3210). I wanted to have a graph-like input and was trying to mimic the behaviour of graph neural networks by masking the tokens (nodes) outside of the neighbourhood. Specifically, if I had for example a graph like `I -> am -> hungry`, and the mapping for tokens was one to one, I would like to have an attention mask like: [[1, 1, 0], [0, 1, 1], [0, 0, 1]], meaning the token `I` would attend to itself and the token `am`, but not `hungry` since it is outside its neighbourhood.  
Is this behaviour possible? I have been searching and found the `get_extended_attention_mask` function. It does work to send a 3D attention mask to the model. However, [this line](https://github.com/huggingface/transformers/blob/94b3f544a1f5e04b78d87a2ae32a7ac252e22e31/src/transformers/modeling_utils.py#L766) says that the 3D attention should be (batch, from\_seq\_len, to\_seq\_len) which suggests that this mask is for cross attention.  
Are `attention_masks` for self-attention or cross-attention?

---

<div class="post-metadata">

**Author:** ![basujindal](https://avatars.discourse-cdn.com/v4/letter/b/d2c977/32.png) [@basujindal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbasujindal)%5C
**Post date:** [June 29, 2024, 8:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoubts-about-attention-masks%2F29356%2F2 "2024-06-29T20:34:07Z")

</div>

HuggingFace is still working on it. There is a PR to support passing custom attention masks. [Allow passing 2D attention mask · Issue #27640 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/27640)