# WordPiece issue - behaves like WordLevel

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwordpiece-issue-behaves-like-wordlevel%2F15973%3E%5C
**Category:** Beginners\
**Created:** [March 22, 2022, 8:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwordpiece-issue-behaves-like-wordlevel%2F15973 "2022-03-22T20:57:43Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![YoelShoshan](https://avatars.discourse-cdn.com/v4/letter/y/f04885/32.png) [@YoelShoshan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FYoelShoshan)%5C
**Post date:** [March 22, 2022, 8:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwordpiece-issue-behaves-like-wordlevel%2F15973%2F1 "2022-03-22T20:57:44Z")

</div>

I don’t know what I’m doing incorrectly, but for some reason, WordPiece behaves like a WordLevel tokenizer.

Note - I do not want to train WordPiece, I have a prebuilt dictionary which I want to use.

Please see a minimalistic code script that reproduces the issue:

```python
from tokenizers import Tokenizer, Regex
from tokenizers.models import WordPiece
from tokenizers.pre_tokenizers import Split 

samples = ['abc','def','abcdef']

model = WordPiece({'abc':10, 'def':20, '<UNK>':100}, unk_token='<UNK>', max_input_chars_per_word = 9999)
tokenizer = Tokenizer(model)

tokenizer.pre_tokenizer = Split(Regex('.*'), behavior='merged_with_previous')

for s in samples:
    print('for input=',s)
    print('standalone pre tekonizer:',tokenizer.pre_tokenizer.pre_tokenize_str(s))
    print('tokenizer output:', tokenizer.encode(s).tokens)
    print('----------------')

```

Which outputs:

```auto
for input= abc
standalone pre tekonizer: [('abc', (0, 3))]
tokenizer output: ['abc']
----------------
for input= def
standalone pre tekonizer: [('def', (0, 3))]
tokenizer output: ['def']
----------------
for input= abcdef
standalone pre tekonizer: [('abcdef', (0, 6))]
tokenizer output: ['<UNK>']

```

I expect the last output to be:  
tokenizer output: [‘abc’, ‘def’]

Any idea what am I doing wrong?  
Any help will be highly appreciated 🙂