# Tokenizing Domain Specific Text
**URL:**
**Author:**  [@jodiak](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjodiak)%5C
**Post date:** [November 11, 2020, 5:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizing-domain-specific-text%2F1978%2F1 "2020-11-11T05:29:49Z")
Hello everyone, Iāve been referencing [this](https://arxiv.org/pdf/2002.00212.pdf) paper on training transformer based models using metadata enhanced MIDI and was thinking about implementing this using the huggingface transformers and tokenizer libraries as an introduction to these libraries beyond the basic language modeling examples. As Iāve been researching and referencing [this](https://huggingface.co/blog/how-to-train) tutorial Iāve ran into issues with tokenization and was wondering when training a tokenizer, how can I set up āword levelā semantics? Technically each āwordā in this case will be the data within a string like so āEvent(name=Position, time=360, value=4/16, text=360)ā rather than just words and characters delimited on spaces like its doing now as listed below
` #version: 0.2 - Trained by huggingface/tokenizers`
`m e`
` a l`
` u e`
` al ue`
` i me`
` n ame`
` v alue`
` Ä value`
` Ä t ex`
` Ä t ime`
` Ev en`
` Ä tex t`
` Even t`
` ) ,`
` Ä Event`
` N o`
Apologies on if these questions are noobish Iām grokking a lot of this as I go along. Any help is greatly appreciated.
---
**Author:**  [@thomwolf](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fthomwolf)%5C
**Post date:** [November 12, 2020, 12:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizing-domain-specific-text%2F1978%2F2 "2020-11-12T12:48:50Z")
pinging @anthony and @Narsil here š
---
**Author:**  [@Narsil](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNarsil)%5C
**Post date:** [November 12, 2020, 1:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizing-domain-specific-text%2F1978%2F3 "2020-11-12T13:18:38Z")
What is your original data like ?
The āWord levelā semantics is usually dealt with the `Pretokenizer` logic (that basically splits up the data where itās relevant). In your case, it would depend on your original data. There is more info in the docs:
> **[The tokenization pipeline ā tokenizers documentation](https://huggingface.co/docs/tokenizers/rust/latest/pipeline.html)**
If you have a specific example of what data comes in, and what you expect as a return, itās probably going to be easier to give you pointers.
Cheers,
Nicolas
---
**Author:**  [@jodiak](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjodiak)%5C
**Post date:** [November 17, 2020, 7:31am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizing-domain-specific-text%2F1978%2F4 "2020-11-17T07:31:40Z")
Hello Narsil, my data looks something similar to this: `Note On_XX`, `Tempo Value_XX`, `Chord_Note:type`, `Position_X/X`, where XX can be a number from 0 to 127 and X can be a number from 1 to 16. I also have a dictionary with every possible values for the above text types numericalized. Will I have to write a custom tokenizer in order to accomplish this tokenization? Is there a way to use my existing numericalized dictionary in a tokenizer? Thank you for your help and the documentation Iāve begun going over it.
---
**Author:**  [@Narsil](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNarsil)%5C
**Post date:** [November 17, 2020, 9:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizing-domain-specific-text%2F1978%2F5 "2020-11-17T09:05:34Z")
Yes it seems you vocabulary is well defined and sufficiently small (127 \* 16 \* 16) to be in a āWordLevelā tokenizer.
> **[Components ā tokenizers documentation](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=wordlevel#models)**
You can even create your vocabulary manually which makes it easier to run.
```python
vocab = {}
i=0
for value in range(0, 128):
for position in range(0, 16):
word = "Note on {value},Position_{position}"
vocab[word] = i
i += 1
import tokenizers
from tokenizers import pre_tokenizers
tokenizer = Tokenizer(BPE(vocab=vocab))
tokenizer.pre_tokenizer = pre_tokenizers.Whitespace()
encoded = tokenizer.encode("Note On_12,Position_16/16 Note on 45,Position_8/8")
encoded.ids # [3, 5]
```
The vocabulary creation I wrote seems a bit off compared to what you described but it should be close enough.
I assumed the possible values are relatively small (\< 50 000) because of 128 notes + 16 tempos + 16 values which makes 32k possible values being conservative. If thereās actually an order of magnitude more, then this approach wonāt work anymore.
Does that help ?
---
**Author:**  [@jodiak](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjodiak)%5C
**Post date:** [November 20, 2020, 4:57am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizing-domain-specific-text%2F1978%2F6 "2020-11-20T04:57:35Z")
Thank you this is incredibly helpful.