# Question about query\_length in modeling\_t5.py

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestion-about-query-length-in-modeling-t5-py%2F16926%3E%5C
**Category:** Beginners\
**Created:** [April 18, 2022, 12:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestion-about-query-length-in-modeling-t5-py%2F16926 "2022-04-18T12:23:44Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![GeneZC](https://avatars.discourse-cdn.com/v4/letter/g/f14d63/32.png) [@GeneZC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGeneZC)%5C
**Post date:** [April 18, 2022, 12:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestion-about-query-length-in-modeling-t5-py%2F16926%2F1 "2022-04-18T12:23:44Z")

</div>

> <https://github.com/huggingface/transformers/blob/0a057201a96565df29984d716f660fd8d634329a/src/transformers/models/t5/modeling_t5.py#L464>

If the `past_key_value` is not None, the `query_length` induced from `SelfAttention` can be the length of `past_key_value` + `hidden_states`, which is actually wrong.

Luckily, this problematic behavior would not affect the `EncDecAttetion` since the `position_bias` are always zeros.

Correct me if I am wrong.