# Help with Tokenizer Word Length Limit
**URL:**
**Author:**  [@smagennis](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsmagennis)%5C
**Post date:** [June 24, 2023, 12:59am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-tokenizer-word-length-limit%2F44364%2F1 "2023-06-24T00:59:52Z")
I’m hitting what seems to me to be an odd limit on the number of characters a Word Piece tokenizer will process before returning [UNK]. I’m working on a project which uses long strings of generated characters that I’m presenting to BERT as a long, ‘strange-looking’ word. Any word less than 100 characters seems to work. At 101 and greater either tokenizer.encode(msg) or tokenizer.\_tokenizer.pre\_tokenizer.pre\_tokenize\_str(msg) seems to give up and return [UNK]. I haven’t been successful tracing down a reason or a way around it. I should be well within the 512 token limit. I welcome any advice or direction. Thanks!
```auto
# word piece tokenizer
tok_ptBert = AutoTokenizer.from_pretrained("bert-base-uncased")
spec_tokens = ['[PAD]','[UNK]', '[MASK]']
trainerX = trainers.WordPieceTrainer(
vocab_size = 20000,
min_frequency = 2,
show_progress=True,
)
# Train tokenizer with new data from tk_train_dataset
tok_ptBert.train_new_from_iterator(tk_train_dataset, 20000)
print('Vocab Size: ' + str(tok_ptBert.vocab_size))
# 100 character long string
msg_100chr_and_below = 'aaaaaaaaaaaaaa .... aaaaaaaaaaaaaaaa'
# 101 character long string
msg_101chr_and_above = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...'
# encode 100 char word
print('raw msg len: ' + str(len(msg_100chr_and_below )))
enc = tok_ptBert.encode(msg_100chr_and_below)
print(enc)
#print(tok_ptBert.tokenize(msg_100chr_and_below))
print(tok_ptBert.convert_ids_to_tokens(enc))
print('===============')
# encode 101(+) char word
print('raw msg len: ' + str(len(msg_101chr_and_above)))
enc = tok_ptBert.encode(msg_101chr_and_above)
print(enc)
#print(tok_ptBert.tokenize(msg_101chr_and_above))
print(tok_ptBert.convert_ids_to_tokens(enc))
Output:
Vocab Size: 30522
raw msg len: 100
[101, 13360, 11057, 11057, [...], 11057, 11057, 11057, 11057, 2050, 102]
['[CLS]', 'aaa', '##aa', '##aa', [...], '##aa', '##aa', '##aa', '##aa', '##a', '[SEP]']
===============
raw msg len: 101
[101, 100, 102]
['[CLS]', '[UNK]', '[SEP]']
```
---
**Author:**  [@mormart](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmormart)%5C
**Post date:** [July 16, 2023, 11:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-tokenizer-word-length-limit%2F44364%2F2 "2023-07-16T11:01:42Z")
I had the same problem. WordPiece tokenizer has an argument `max_input_chars_per_word`, see [here](https://huggingface.co/docs/tokenizers/api/models#tokenizers.models.WordPiece). The default is [set to 100](https://github.com/huggingface/tokenizers/blob/4811f769a1477f99283ffe95837a0fc2c3aeeb3c/tokenizers/src/models/wordpiece/mod.rs#L49C3-L49C3) in the rust implementation.
You can change this to a value larger than your max number of characters per ‘word’, e.g.:
```auto
>>> from transformers import AutoTokenizer
>>> tok_ptBert = AutoTokenizer.from_pretrained("bert-base-uncased")
>>> tok_ptBert.tokenize('a' * 101)
['[UNK]']
>>> tok_ptBert._tokenizer.model.max_input_chars_per_word = 1000
>>> tok_ptBert.tokenize('a' * 101)
['aaa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa', '##aa']
```
---
**Author:**  [@smagennis](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsmagennis)%5C
**Post date:** [July 16, 2023, 6:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-tokenizer-word-length-limit%2F44364%2F3 "2023-07-16T18:50:10Z")
Brilliant! Thanks mormart!