# Use a pretrained ByteLevelBPETokenizer on text

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fuse-a-pretrained-bytelevelbpetokenizer-on-text%2F348%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [July 17, 2020, 7:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fuse-a-pretrained-bytelevelbpetokenizer-on-text%2F348 "2020-07-17T07:52:26Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![abdallah197](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/abdallah197/32/3973_2.png) [@abdallah197](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabdallah197)%5C
**Post date:** [July 17, 2020, 7:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fuse-a-pretrained-bytelevelbpetokenizer-on-text%2F348%2F1 "2020-07-17T07:52:26Z")

</div>

Hi  
I am asking whether there’s a simple way to tokenize a piece of text “I will go to the bedroom” to BPE " I will go to the bed ##room" without training a tokenizer from scratch.

---

<div class="post-metadata">

**Author:** ![anthony](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anthony/32/2505_2.png) [@anthony](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fanthony)%5C
**Post date:** [July 17, 2020, 2:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fuse-a-pretrained-bytelevelbpetokenizer-on-text%2F348%2F2 "2020-07-17T14:20:26Z")

</div>

Hi @abdallah197!

There is a bunch of pre-trained tokenizers in the [huggingface/transformers](https://github.com/huggingface/transformers) library that you can use directly, without having to train anything. You won’t have any control over how the tokens are split though, as this is based on what the tokenizer learned during training, and the size of its vocabulary. `bedroom` isn’t really a rare word, so often it will have its own token in the vocabulary.

Your example looks like a WordPiece (instead of BPE), given the `##` in `##room` which is very specific to this kind of tokenizers. You can try to use those from `BERT` in the library to see if anything fits your needs, for example with:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

tokenizer.tokenize("I will go to the bedroom")
# ['I', 'will', 'go', 'to', 'the', 'bedroom']

tokenizer.tokenize("I will go to the Bedroom")
# ['I', 'will', 'go', 'to', 'the', 'Bed', '##room']

```