# Pad Tokens & Attention Masks with Data Collators

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpad-tokens-attention-masks-with-data-collators%2F104465%3E%5C
**Category:** 🤗Transformers\
**Created:** [August 29, 2024, 8:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpad-tokens-attention-masks-with-data-collators%2F104465 "2024-08-29T08:19:23Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Chahnwoo](https://avatars.discourse-cdn.com/v4/letter/c/a88e57/32.png) [@Chahnwoo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FChahnwoo)%5C
**Post date:** [August 29, 2024, 8:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpad-tokens-attention-masks-with-data-collators%2F104465%2F1 "2024-08-29T08:19:23Z")

</div>

I have a few quick questions about how the HuggingFace DataCollator classes (DataCollatorForSeq2Seq in particular) process pad tokens.

Data Format:

```auto
{
        "input_ids" : [1, 2, 3, 4, 5, 6, ..., 2],
        "attention_masks" : [1, 1, 1, 1, 1, 1, ..., 1],
        "labels" : [-100, -100, -100, -100, -100, -100, ..., 2]
}

```

1. When data of the above format is passed to the DataCollatorForSeq2Seq and padding is performed, does the data collator also automatically add 0’s to the attention mask to indicate that the padding data should be ignored in the model’s attention mechanisms?
2. If the data collator does indeed perform the above, does this remain true models where the `pad_token` is not defined (e.g. LLaMA-3.1-8B)?