")
# and save for later!
tokenizer.save_pretrained("./path/to/transformers/version/")
```
This seems to load properly using `AutoTokenizer.from_pretrained()` which should make life a lot easier.
---
**Author:**  [@appledora](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fappledora)%5C
**Post date:** [January 16, 2023, 8:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-sentencepiece-from-scratch%2F3477%2F5 "2023-01-16T20:56:16Z")
Thanks for this code snippet! How did you format the dataset (`text`) here? One sentence per line? I was also wondering whether I could train it iteratively for multiple chunks of text input?
---
**Author:**  [@jbmaxwell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjbmaxwell)%5C
**Post date:** [January 16, 2023, 9:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-sentencepiece-from-scratch%2F3477%2F6 "2023-01-16T21:24:23Z")
You’re welcome! I know it always helps to see code.
In my case it was just one line per sentence in a single flat text file.
---
**Author:**  [@derVokler](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FderVokler)%5C
**Post date:** [February 1, 2023, 2:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-sentencepiece-from-scratch%2F3477%2F8 "2023-02-01T14:23:47Z")
Hi everyone,
I found this thread when I was looking for information how to best use a custom Sentencepiece model with the huggingface models / tokenizers. I found a different solution that works for me:
1. Train a Sentencepiece model with the Sentencepiece library
2. Load it one time into the tokenizer that I want
3. Save that tokenizer with .save\_pretrained()
After that it can be loaded with .from\_pretrained(). Here are the steps with a little more detail.
First, training the Sentencepiece model:
```auto
import sentencepiece as spm
spm.SentencePieceTrainer.Train(
input=,
model_prefix='spModel',
vocab_size=1000,
pad_id=0,
unk_id=1,
bos_id=2,
eos_id=3,
pad_piece='[PAD]',
unk_piece='[UNK]',
bos_piece='[CLS]',
eos_piece='[SEP]',
user_defined_symbols='[MASK]',
model_type='unigram'
)
```
Now load it into a tokenizer, here a DebertaV2Tokenizer (why this needs to be the vocab file, I am not sure, but it works; I also specify the maximum length here):
```auto
from transformers import DebertaV2Tokenizer
tokenizer_deberta = DebertaV2Tokenizer(
vocab_file = "spModel.model",
max_len = 512,
)
```
Now save as a pretrained tokenizer:
```auto
tokenizer_deberta.save_pretrained( PATH )
```
And from that point on you can load it as any pretrained tokenizer:
```auto
tokenizer_loaded = DebertaV2Tokenizer.from_pretrained(
PATH
)
```
When I print that guy, it looks to me like all special tokens and the sequence length are correct:
> DebertaV2Tokenizer(name\_or\_path=PATH, vocab\_size=1000, model\_max\_length=512, is\_fast=False, padding\_side=‘right’, truncation\_side=‘right’, special\_tokens={‘bos\_token’: ‘[CLS]’, ‘eos\_token’: ‘[SEP]’, ‘unk\_token’: ‘[UNK]’, ‘sep\_token’: ‘[SEP]’, ‘pad\_token’: ‘[PAD]’, ‘cls\_token’: ‘[CLS]’, ‘mask\_token’: ‘[MASK]’})
Maybe this still helps. I think the nice thing about this approach is that you can get a custom _unigram_ tokenizer.
---
**Author:**  [@VS9205](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FVS9205)%5C
**Post date:** [November 5, 2023, 11:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-sentencepiece-from-scratch%2F3477%2F9 "2023-11-05T11:05:31Z")
How did you format the dataset (`text` ) here?
---
**Author:**  [@jubueche](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjubueche)%5C
**Post date:** [December 19, 2023, 10:23am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-sentencepiece-from-scratch%2F3477%2F10 "2023-12-19T10:23:53Z")
I couldn’t figure out how to pass an iterator to this function. Do you know how to do that? Otherwise one always has to create a file first.