# Unigram vocab\_size doesn't fit

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funigram-vocab-size-doesnt-fit%2F26856%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [November 28, 2022, 11:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funigram-vocab-size-doesnt-fit%2F26856 "2022-11-28T11:06:40Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Axllm](https://avatars.discourse-cdn.com/v4/letter/a/c5a1d2/32.png) [@Axllm](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAxllm)%5C
**Post date:** [November 28, 2022, 11:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funigram-vocab-size-doesnt-fit%2F26856%2F1 "2022-11-28T11:06:40Z")

</div>

Hi there, thanks for your amazing work !

I encounter an issue when I’m using Unigram and UnigramTrainer.  
In fact I want to build a vocabulary of size 10 (based on vocabulary built with BPE)

But I don’t understand why, using both train\_from\_iterator (with list and with iterator) and with train (from file name containing same information as in list) I can’t retrieve a vocab of wished size.

Here is some code to work with:

```auto
corpus = ["table", "bleu", "cable"]
# tokenizers. __version__ == 0.13.2
from tokenizers import Tokenizer
from tokenizers.models import Unigram

vocab = ["a", "b", "c", "e", "l", "t", "u", "bl", "ble", "able", "cable"]
tokenizer = Tokenizer(Unigram())
tokenizer.add_tokens(vocab)
print("With added tokens")
print(tokenizer.get_vocab(with_added_tokens=True))
print(tokenizer.get_vocab_size(with_added_tokens=True))
print("Without added tokens")
print(tokenizer.get_vocab(with_added_tokens=False))
print(tokenizer.get_vocab_size(with_added_tokens=False))

from tokenizers.trainers import UnigramTrainer
trainer = UnigramTrainer(vocab_size=10)#, initial_alphabet=["a", "b", "c", "e", "l", "t", "u"])
tokenizer.train_from_iterator(corpus, trainer=trainer)
#tokenizer.train(["corpus.txt"], trainer=trainer)
#tokenizer.train_from_iterator(iterator(), trainer=trainer)

print("With added tokens")
print(dict(sorted(tokenizer.get_vocab(with_added_tokens=True).items(), key=lambda item: item[1])))
print(tokenizer.get_vocab_size(with_added_tokens=True) )

print("Without added tokens")
print(dict(sorted(tokenizer.get_vocab(with_added_tokens=False).items(), key=lambda item: item[1])))
print(tokenizer.get_vocab_size(with_added_tokens=False) )

```

And results associated :

 ![Capture d’écran du 2022-11-28 12-05-57](https://us1.discourse-cdn.com/hellohellohello/original/2X/9/9375d2c678ad38d2585545837382887744d2a6c8.png)