"] + optional_specials
trainer = BpeTrainer(special_tokens=special_tokens, end_of_word_suffix="", vocab_size=4000)
tokenizer.pre_tokenizer = Whitespace()
tokenizer.save(f'./tokenizer/{filter}_bpe_tokenizer/tokenizer.json')
```
(I was originally using BERT-like special tokens ("[CLS]", “[SEP]”, etc.), but switched to the GPT-like ones as a test. Either way the spaces are missing.)
I load to a Transformers tokenizer using `PreTrainedTokenizerFast`:
```auto
fast_tokenizer = PreTrainedTokenizerFast(tokenizer_file=f"./tokenizer/{filter}_bpe_tokenizer/tokenizer.json")
```
When I try to decode:
`decode_test = fast_tokenizer.decode(test)`
the result is correct except that there are no spaces after any of my `optional_specials`, e.g.:
`... [BL]MD3a6 [BT1]CCHHr6 DM100 ST27 ST34 ST36 ...`
instead of:
`... [BL] MD3a6 [BT1] CCHHr6 DM100 ST27 ST34 ST36 ...`
This looks like maybe it’s intending to remove special tokens in the decoded version, but I’m not sure. How do I fix this? I want the `optional_specials` in the output because I have another process that uses them to segment the result.
---
**Author:**  [@jbmaxwell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjbmaxwell)%5C
**Post date:** [April 5, 2022, 10:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbpedecoder-no-spaces-after-special-tokens%2F16511%2F2 "2022-04-05T22:58:57Z")
I’ve worked around it by removing the Tokenizers `decoder` and instead using:
```auto
tokenizer = Tokenizer(BPE(unk_token="[UNK]", continuing_subword_prefix="##"))
```
That seems to be giving me what I’d expect. Maybe there’s something funky about the way the Tokenizers and Transformers libs deal with decoders… Anyway, this seems like it will work for what I need.
---
**Author:**  [@jbmaxwell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjbmaxwell)%5C
**Post date:** [April 5, 2022, 11:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbpedecoder-no-spaces-after-special-tokens%2F16511%2F3 "2022-04-05T23:09:36Z")
but… I’m noticing that the saved `tokenizer.json` shows `"continuing_subword_prefix": null,`… ☹
no idea
If this doesn’t work to avoid having a bunch of white space in my output words, then I’ll go back to the decoder-based version and just post-process the lack of spaces between specials and subsequent words.
---
**Author:**  [@Davidg707](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDavidg707)%5C
**Post date:** [January 2, 2023, 4:22am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbpedecoder-no-spaces-after-special-tokens%2F16511%2F4 "2023-01-02T04:22:14Z")
I’m noticing too that `tokenizer.train_from_iterator` seems to erase the `continuing_subword_prefix` setting. Did you ever solve this?
For the HF devs, to replicate:
```python
wiki103 = datasets.load_dataset(
path="wikitext",
name="wikitext-103-raw-v1",
split="train",
)
tokenizer = Tokenizer(
model=BPE(
unk_token="[UNK]",
continuing_subword_prefix="##",
)
)
trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])
tokenizer.pre_tokenizer = Whitespace()
assert tokenizer.model.continuing_subword_prefix == "##" # Fine
tokenizer.train_from_iterator(
iterator=wiki103["text"],
trainer=trainer,
)
assert tokenizer.model.continuing_subword_prefix == "##" # Assertion Error
```
---
**Author:**  [@keunwoochoi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkeunwoochoi)%5C
**Post date:** [April 19, 2023, 5:08am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbpedecoder-no-spaces-after-special-tokens%2F16511%2F5 "2023-04-19T05:08:00Z")
having the same issue after `tokenizer.train_from_iterator()`.