# Sentenspeace tokenizer or mbart50

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsentenspeace-tokenizer-or-mbart50%2F151125%3E%5C
**Category:** Beginners\
**Created:** [April 18, 2025, 11:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsentenspeace-tokenizer-or-mbart50%2F151125 "2025-04-18T11:28:05Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![ahmed1488](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ahmed1488/32/45790_2.png) [@ahmed1488](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fahmed1488)%5C
**Post date:** [April 18, 2025, 11:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsentenspeace-tokenizer-or-mbart50%2F151125%2F1 "2025-04-18T11:28:05Z")

</div>

hi , i m making a seq2seq model with attention for multilinguage translation , i trained a sentenspeace tokenizer on my data but after training when passing unlearned sentences for translation i got this poor translation:

Source: good morning  
Translation: bonne

Source: give me your email  
Translation: donne-moi ton email

Source: give me your email please  
Translation: donne-moi ton e sil vous plaît

i m using sentencepeace tokenizer like this import sentencepiece as spm

# Train the SentencePiece tokenizer

spm.SentencePieceTrainer.train(  
input=‘spm\_training\_data.txt’,  
model\_prefix=‘multilingual\_tokenizer’,  
vocab\_size=32000,  
character\_coverage=0.9995,  
model\_type=‘bpe’,  
bos\_id=1,  
eos\_id=2,  
pad\_id=0,  
unk\_id=3,  
user\_defined\_symbols=[“\<en\_US\>”, “\<fr\_FR\>”],  
split\_by\_whitespace=True  
)

# Load the trained tokenizer

sp = spm.SentencePieceProcessor()  
sp.load(‘multilingual\_tokenizer.model’)

should i used a pretrained tokenizer like mbart50 tokenizer, is it ok to use a pretrained tokenizer for my  
own build seq2seq model with attention .i am concerned about the huge vocab size of 250k,what should i do .