# Attention mask shape (custom attention masking)
**URL:**
**Author:**  [@alex-hh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Falex-hh)%5C
**Post date:** [August 27, 2024, 8:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fattention-mask-shape-custom-attention-masking%2F104272%2F1 "2024-08-27T20:53:25Z")
In e.g. the [LlamaModel docs](https://huggingface.co/docs/transformers/main/en/model_doc/llama2#transformers.LlamaModel.forward) it suggests that the attention\_mask passed to forward should be 2 dimensional.
However looking at the source code, it looks like it is possible to provide a 4D mask, and this will override the standard (e.g. causal) mask.
Is this correct? Should this be documented? Is there anything to watch out for when doing this? (I’m interested in providing a custom attention masking pattern to the Llama architecture).
---
**Author:**  [@AnneTheBoo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAnneTheBoo)%5C
**Post date:** [February 17, 2025, 3:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fattention-mask-shape-custom-attention-masking%2F104272%2F2 "2025-02-17T15:38:19Z")
> [@alex-hh](#):
>
> Is this correct? Should this be documented? Is there anything to watch out for when doing this? (I’m interested in providing a custom attention masking pattern to the Llama architecture).
Hi,
yes I passed a custom mask in 4D and it worked 🙂 (confirmed with the attention scores)
Relating to your question - did you perhaps find out how to answer this question regarding additive/binary attention mask?
> [@Does Llama-2 use additive attention masking?](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-llama-2-use-additive-attention-masking%2F140399)%3A
>
> Hi, I need to build a custom causal attention mask for meta-llama/Llama-2-7b-chat-hf from LlamaForCausalML. Can someone confirm that it uses additive masking (0 or large negative number close to -inf) instead of binary masking (0 or 1) for its attention mechanism? I found that binary masking doesn’t work when I investigated the attention scores, whereas additive masking actually lead to attention scores being 0 in the right dimensions. But on the other hand the documentation says it uses bina…
---
**Author:**  [@spranav1205](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fspranav1205)%5C
**Post date:** [April 5, 2025, 1:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fattention-mask-shape-custom-attention-masking%2F104272%2F3 "2025-04-05T13:14:35Z")
Hey,
I want to pass a custom mask, too. any chance you can help me out?
How did you pass it?
(Again, to override the standard causal mask)
Thanks in advance
---
**Author:**  [@AnneTheBoo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAnneTheBoo)%5C
**Post date:** [April 27, 2025, 10:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fattention-mask-shape-custom-attention-masking%2F104272%2F4 "2025-04-27T10:46:53Z")
Hi @spranav1205,
so I am using LlamaForCausalML, here is my code snippet:
```auto
def generate_additive_attention_mask(no_masking_length, total_length):
# no_masking_length is the number of tokens you don't need masking for
mask = torch.tril(torch.ones(total_length, total_length)).to(device)
mask[:no_masking_length, :no_masking_length] = 1
mask = mask.unsqueeze(0).unsqueeze(0) # Add batch and num_attention_heads dimensions
mask = (1 - mask) * -1e9 # Because Llama2.7b-chat uses additive masking, set to -large number instead of 0, and set to 0 instead of 1.
inputs = tokenizer(msg)
seq_len = inputs["input_ids"].shape[1]
inputs["attention_mask"]=generate_additive_attention_mask(no_masking_length, seq_len)
outputs = model(**inputs)
```