# T5 tokenizer vs t51.1 tokenizer

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-tokenizer-vs-t51-1-tokenizer%2F75421%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [March 1, 2024, 2:35am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-tokenizer-vs-t51-1-tokenizer%2F75421 "2024-03-01T02:35:55Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nevermetyou](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nevermetyou/32/21618_2.png) [@Nevermetyou](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNevermetyou)%5C
**Post date:** [March 1, 2024, 2:35am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-tokenizer-vs-t51-1-tokenizer%2F75421%2F1 "2024-03-01T02:35:55Z")

</div>

Hello, there

I wonder if the tokenizer of t51.1 is the same as that of normal t5.  
 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/0/c/0c9790f4600c4434b458f1dbb6cdd87ea049f070.png)

Or is the pre-tokenization step not the same?