# Documentation of SentencePieceBPETokenizer?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdocumentation-of-sentencepiecebpetokenizer%2F84777%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [May 2, 2024, 6:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdocumentation-of-sentencepiecebpetokenizer%2F84777 "2024-05-02T06:48:48Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![raptorkwok](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/raptorkwok/32/26481_2.png) [@raptorkwok](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fraptorkwok)%5C
**Post date:** [May 2, 2024, 6:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdocumentation-of-sentencepiecebpetokenizer%2F84777%2F1 "2024-05-02T06:48:48Z")

</div>

I cannot find `SentencePieceBPETokenizer` in the official documentation, therefore I ask this question.

I refer to the [source code](https://github.com/huggingface/tokenizers/blob/f2ec3b239b0a7a9866b01ec5cbd4d44243a40a16/bindings/python/py_src/tokenizers/implementations/sentencepiece_bpe.py#L78) of `SentencePieceBPETokenizer` to write my codes to train a custom Tokenizer based on `SentencePieceBPETokenizer`.

**Question 1:** do the following codes make sense?

```auto
from tokenizers import SentencePieceBPETokenizer

special_tokens = ["<unk>", "<pad>", "<cls>", "<sep>", "<mask>"]
tokenizer = SentencePieceBPETokenizer()
tokenizer.train_from_iterator(
    get_training_corpus(),
    vocab_size = 400000,
    min_frequency = 10, 
    show_progress = True,
    special_tokens = special_tokens
)

```

where `get_training_corpus()` is a function to get sentences in batch from a list of strings.

I notice that there are `Trainer` classes, but there is no `SentencePiece`. Also, from the `Tokenizer` [documentation](https://huggingface.co/docs/tokenizers/api/tokenizer#tokenizers.Tokenizer.train_from_iterator), it specifies the attributes of `train_from_iterator()` are `iterator` and `trainer` object, but my attributes follow the `Trainer` class instead.

**Question 2:** No error occurred, can I assume the attributes are correct?

**Question 3:** What is the use of `initial_alphabet` and `limit_alphabet`? The documentation does not tell clearly. Any example?

p.s. I know `show_progress = True` does not work in Jupyter notebook.  
p.s. using `tokenizers 0.13.3`