# Space token ' ' cannot be add when is\_split\_into\_words = True

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspace-token-cannot-be-add-when-is-split-into-words-true%2F4305%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [March 11, 2021, 9:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspace-token-cannot-be-add-when-is-split-into-words-true%2F4305 "2021-03-11T09:27:29Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Boltzmachine](https://avatars.discourse-cdn.com/v4/letter/b/46a35a/32.png) [@Boltzmachine](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBoltzmachine)%5C
**Post date:** [March 11, 2021, 9:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspace-token-cannot-be-add-when-is-split-into-words-true%2F4305%2F1 "2021-03-11T09:27:29Z")

</div>

for example,

```python
>>> tokenizer = BertTokenizer.from_pretrained('bert-base-chinese')
>>> tokenizer.add_tokens(' ')
1

```

```python
>>> tokenizer.encode('你好 世界', add_special_tokens=False)
[872, 1962, 21128, 686, 4518]
>>> tokenizer.encode(['你','好',' ', '世', '界'], is_split_into_words=True, add_special_tokens=False)
 [872, 1962, 686, 4518]

```

Obviously, the blank token is ignored. But if you change it to another token like ‘[balabala]’, it works.  
So what is the proper way to do this?

---

<div class="post-metadata">

**Author:** ![Boltzmachine](https://avatars.discourse-cdn.com/v4/letter/b/46a35a/32.png) [@Boltzmachine](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBoltzmachine)%5C
**Post date:** [March 11, 2021, 9:35am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspace-token-cannot-be-add-when-is-split-into-words-true%2F4305%2F2 "2021-03-11T09:35:35Z")

</div>

I found that one way is to use `convert_tokens_to_ids`, yet by which I cannot use the convenient features in `encode` and ` __call__ ` such as padding and automatically generating attention\_mask