# Self-attention query vs key size in gpt2

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fself-attention-query-vs-key-size-in-gpt2%2F19226%3E%5C
**Category:** 🤗Transformers\
**Created:** [June 16, 2022, 5:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fself-attention-query-vs-key-size-in-gpt2%2F19226 "2022-06-16T05:17:30Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![amanrs](https://avatars.discourse-cdn.com/v4/letter/a/8edcca/32.png) [@amanrs](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Famanrs)%5C
**Post date:** [June 16, 2022, 5:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fself-attention-query-vs-key-size-in-gpt2%2F19226%2F1 "2022-06-16T05:17:30Z")

</div>

For non-cross-attention (self-attention), I’d expect the query and key size to always be the same.

> <https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt2/modeling_gpt2.py#L200-L201>

Is there a reason why they could be different sizes here?

---

<div class="post-metadata">

**Author:** ![dblakely](https://avatars.discourse-cdn.com/v4/letter/d/54ee81/32.png) [@dblakely](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdblakely)%5C
**Post date:** [June 17, 2022, 9:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fself-attention-query-vs-key-size-in-gpt2%2F19226%2F2 "2022-06-17T21:02:41Z")

</div>

The query length is typically 1, as it corresponds to the most recently predicted token being fed through the model attending to the past tokens (and itself). The keys and values, on the other hand, will be \> 1, as they correspond to the past tokens that have already been written. The keys and values are always the same length.

There are cases where all three of them will have the same length. Namely, when you pass a set of `input_ids` to the model–for one timestep, they’ll all be the same length. Then the query length will be back to 1 afterwards.