# Inference input token number set as the max length always?
**URL:**
**Author:**  [@kaixin123](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkaixin123)%5C
**Post date:** [April 21, 2024, 3:40pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-input-token-number-set-as-the-max-length-always%2F82939%2F1 "2024-04-21T15:40:17Z")
I am a littile confused. In transformer based LLM inference. We always set the input length as the max\_input length and padding 0 , or we dynamically get the actual input length and decrease the computation?
---
**Author:**  [@vivek9840](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvivek9840)%5C
**Post date:** [April 21, 2024, 7:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-input-token-number-set-as-the-max-length-always%2F82939%2F2 "2024-04-21T19:26:03Z")
it depends upon how you are inferring LLMs. while batching you pad it to the length of longest sequence in the batch. for accelerators, you might pad it to longest sequence and multiple of 8. and while single input inferencing you just pass it without padding. just to make sure that in text-generation, you pad to the left. since models might get hard time while start generation after pad token. but sometimes they have right side padding, like codellama. about the padding side. about padding side i wish i knew earlier just that.
---
**Author:**  [@kaixin123](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkaixin123)%5C
**Post date:** [April 21, 2024, 7:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-input-token-number-set-as-the-max-length-always%2F82939%2F3 "2024-04-21T19:44:43Z")
Thanks for your reply. Let’s say in chat server, such as GPT or huggingchat. They will always pad to the max\_length or just pass it without padding? I ask this question since I wanna understand efficient inference diving into the hardware level. Is it difficult to design the pattern just dynamically detect each time’s input tokens length and pass it without padding?
---
**Author:**  [@vivek9840](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvivek9840)%5C
**Post date:** [April 21, 2024, 7:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-input-token-number-set-as-the-max-length-always%2F82939%2F4 "2024-04-21T19:59:47Z")
by default it always passes the sequence without padding. i am not sure about GPT but in general, if you are inferring it with single sequence. it does not pad. but it might be different for devices which compile model for XLA. in that case you need to pad, such that the model does not have to compile again and again.
---
**Author:**  [@kaixin123](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkaixin123)%5C
**Post date:** [April 21, 2024, 8:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-input-token-number-set-as-the-max-length-always%2F82939%2F5 "2024-04-21T20:16:32Z")
Thanks again! That make sense. I’m not familiar with XLA. But if this is similar to CUBLAS etc, then I think there is a trade-off between the compile time and the extra-padding computation time, right? Is this true for GPU? If we passes without padding, then we need to compile every time when input is different?
---
**Author:**  [@kaixin123](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkaixin123)%5C
**Post date:** [April 21, 2024, 8:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-input-token-number-set-as-the-max-length-always%2F82939%2F6 "2024-04-21T20:18:43Z")
By the way, what do you mean single sequence? Does it mean we only input one question into our model and do the inference? In other words, If consecutive questions, we need to set fix length?
---
**Author:**  [@vivek9840](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvivek9840)%5C
**Post date:** [April 21, 2024, 8:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-input-token-number-set-as-the-max-length-always%2F82939%2F7 "2024-04-21T20:19:49Z")
i am not sure about GPU, since when i use batching i pad it to maximum sequence and while single inference, i just don’t change it. for TPU, it is specifically mentioned in the introduction. so that’s how i know. i am not sure about GPU.
---
**Author:**  [@kaixin123](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkaixin123)%5C
**Post date:** [April 21, 2024, 8:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-input-token-number-set-as-the-max-length-always%2F82939%2F8 "2024-04-21T20:23:37Z")
Thanks! Could I ask the reason why batching should pad it to maximum sequence?
---
**Author:**  [@vivek9840](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvivek9840)%5C
**Post date:** [April 21, 2024, 8:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-input-token-number-set-as-the-max-length-always%2F82939%2F9 "2024-04-21T20:26:35Z")
because, if we don’t pad it. it truncate to shortest one. which make cut of all the remaining sequences which are longer than shortest sequence. which is loss in input of remaining sequence and cause output which is not expected.
---
**Author:**  [@kaixin123](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkaixin123)%5C
**Post date:** [April 21, 2024, 8:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-input-token-number-set-as-the-max-length-always%2F82939%2F10 "2024-04-21T20:27:16Z")
Oh, I understand, thanks!
---
**Author:**  [@vivek9840](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvivek9840)%5C
**Post date:** [April 21, 2024, 8:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-input-token-number-set-as-the-max-length-always%2F82939%2F11 "2024-04-21T20:28:39Z")
> [@Purpose of padding and truncating](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpurpose-of-padding-and-truncating%2F412)%3A
>
> I have read the [Preprocessing Data page](https://huggingface.co/transformers/preprocessing.html#base-use). I understand what padding and truncating are doing, but I’m not sure I understand the reason for doing either of them. Can anyone help me understand the purpose for doing them? Thanks in advance!
> **[Padding and truncation](https://huggingface.co/docs/transformers/en/pad_truncation)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
these are the sources which i learned from. might help you.