# wordLevel tokenizing

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwordlevel-tokenizing%2F95430%3E%5C
**Category:** Beginners\
**Created:** [July 4, 2024, 1:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwordlevel-tokenizing%2F95430 "2024-07-04T01:04:26Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![farshidk](https://avatars.discourse-cdn.com/v4/letter/f/e274bd/32.png) [@farshidk](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffarshidk)%5C
**Post date:** [July 4, 2024, 1:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwordlevel-tokenizing%2F95430%2F1 "2024-07-04T01:04:26Z")

</div>

hey guys,  
I want to tokenize a DNA seq. it’s like [[AAA GGG TTT],[AGT ATT CGC CCC AAA GTT],…]  
as you see it encompasses different sequences with different lengths, generally we have 64 unique codons (each of the elements in our list(for example AAA) is a codon)  
how can I tokenize in word level that separated only in spaces.  
also I want to padding and truncating my tokenized list (with input\_ids, mask columns that every tokenized file in hugging face pretrained library has

somehow this code works but it was not whta i want  
from tokenizers import Tokenizer  
from tokenizers.models import WordLevel  
from tokenizers.trainers import WordLevelTrainer  
from tokenizers.pre\_tokenizers import Whitespace  
from transformers import PreTrainedTokenizerFast

# Create the tokenizer

tokenizer = Tokenizer(WordLevel(unk\_token=“[UNK]”))  
tokenizer.pre\_tokenizer = Whitespace()

# Define special tokens

special\_tokens\_dict = {‘cls\_token’: ‘[CLS]’, ‘sep\_token’: ‘[SEP]’, ‘pad\_token’: ‘[PAD]’, ‘mask\_token’: ‘[MASK]’}

# tokenizer.add\_special\_tokens(special\_tokens\_dict)

# Define the trainer with special tokens

trainer = WordLevelTrainer(special\_tokens=list(special\_tokens\_dict.values()))

# Example training data

training\_data = s.DNA\_Codons.keys()

# Save the training data to a temporary file

with open(“training\_data.txt”, “w”) as f:  
f.write(“\n”.join(training\_data))

# Train the tokenizer

tokenizer.train([“training\_data.txt”], trainer)

# Save the tokenizer to a file

tokenizer.save(“C:\Users\farsh\Downloads\tokenizer\_DNA.json”)