# How are the inputs tokenized when model deployment?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-are-the-inputs-tokenized-when-model-deployment%2F9692%3E%5C
**Category:** Amazon SageMaker\
**Created:** [September 2, 2021, 1:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-are-the-inputs-tokenized-when-model-deployment%2F9692 "2021-09-02T13:39:10Z")\
**Posts on this page:** 1\
**Showing post:** 12

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fphilschmid)%5C
**Post date:** [September 3, 2021, 9:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-are-the-inputs-tokenized-when-model-deployment%2F9692%2F12 "2021-09-03T09:13:44Z")

</div>

> [@Oigres](#):
>
> `Token indices sequence length is longer than the specified maximum sequence length for this model (520 > 512). Running this sequence through the model will result in indexing errors"`

The Error is pretty clear. It looks like `truncate` is not applied correctly.  
Which model are you using?

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-are-the-inputs-tokenized-when-model-deployment%2F9692)._