# WordPiece tokenizer doesn't work for long sequences

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwordpiece-tokenizer-doesnt-work-for-long-sequences%2F27391%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [December 7, 2022, 12:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwordpiece-tokenizer-doesnt-work-for-long-sequences%2F27391 "2022-12-07T12:08:24Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![dotan1111](https://avatars.discourse-cdn.com/v4/letter/d/85f322/32.png) [@dotan1111](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdotan1111)%5C
**Post date:** [December 7, 2022, 12:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwordpiece-tokenizer-doesnt-work-for-long-sequences%2F27391%2F1 "2022-12-07T12:08:24Z")

</div>

I’m trying to train a WordPiece (WPC) tokenizer from HuggingFace on long sequences. I know the tokenizer is created successfully by looking at the saved file. When I’m trying to encode a new sequences, the tokenizer return only unknow tokens. Once shortening the sequence, the tokenizer return a valid tokens. When encoding the same sequences with a different type of tokenizer (Unigram or BPE) the tokenizers returns valid results. I’m not getting any errors or warning from the library.

---

<div class="post-metadata">

**Author:** ![lkurlandski](https://avatars.discourse-cdn.com/v4/letter/l/b782af/32.png) [@lkurlandski](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flkurlandski)%5C
**Post date:** [March 28, 2024, 1:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwordpiece-tokenizer-doesnt-work-for-long-sequences%2F27391%2F2 "2024-03-28T13:58:00Z")

</div>

I have encountered the exact same issue with tokenizers==0.14.1.