# Preprocessing data for custom tokenizer

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpreprocessing-data-for-custom-tokenizer%2F24767%3E%5C
**Category:** 🤗Transformers\
**Created:** [October 21, 2022, 9:12am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpreprocessing-data-for-custom-tokenizer%2F24767 "2022-10-21T09:12:43Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![antoine2323231](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@antoine2323231](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fantoine2323231)%5C
**Post date:** [October 21, 2022, 9:12am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpreprocessing-data-for-custom-tokenizer%2F24767%2F1 "2022-10-21T09:12:43Z")

</div>

I am training a custom tokenizer on is one long string. Can I parse it directly to a tokenizer model and it would use the EOS token when it is a “.” for example so that it understand the sentence transition? My question is: I do not need to split it into lines, correct?