# WordLevel Tokenization with GPT2?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwordlevel-tokenization-with-gpt2%2F5153%3E%5C
**Category:** 🤗Transformers\
**Created:** [March 27, 2021, 6:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwordlevel-tokenization-with-gpt2%2F5153 "2021-03-27T18:30:44Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Athena](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/athena/32/1417_2.png) [@Athena](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAthena)%5C
**Post date:** [March 27, 2021, 6:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwordlevel-tokenization-with-gpt2%2F5153%2F1 "2021-03-27T18:30:44Z")

</div>

I am thinking of pretraining a GPT2 model from scratch where the tokens must be of whole words (e.g. using the `WordLevel` tokenizer) instead of subwords (e.g. using the `ByteLevelBPETokenizer` tokenizer). This is because subwords don’t make sense for my application.

Will the `GPT2LMHeadModel` and `GPT2Tokenizer` be able to accept word level tokenization?

Because a `Tokenizer(WordLevel())` tokenizer [saves to a single json file](https://huggingface.co/docs/tokenizers/python/latest/api/reference.html#tokenizers.Tokenizer.save)

```auto
tokenizer = Tokenizer(WordLevel())
tokenizer.train(files=["words.txt"])
tokenizer.save('wordlevel.json')

```

but the `GPT2Tokenizer` [reads from 2 files](https://huggingface.co/transformers/model_doc/gpt2.html#transformers.GPT2Tokenizer) and is based on a byte-level Byte-Pair-Encoding.

```auto
tokenizer = GPT2Tokenizer(
    os.path.join("vocab.json"),
    os.path.join("merges.txt"),
)

```

If byte-level byte-pair encoding must be used, can `ByteLevelBPETokenizer` be configured to do word level tokenization?

Thanks 🙂

---

<div class="post-metadata">

**Author:** ![boranapak](https://avatars.discourse-cdn.com/v4/letter/b/839c29/32.png) [@boranapak](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fboranapak)%5C
**Post date:** [March 26, 2023, 1:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwordlevel-tokenization-with-gpt2%2F5153%2F2 "2023-03-26T13:01:33Z")

</div>

Hey Athena,

I also want to generate tokens that must be whole words and I have the same problem you had 2 years ago! Did you end up figuring out a way to solve this problem? I know it has been a long time but it would really help me out a lot!

Thanks in advance!