# Dealing with Decimal and Fractions

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdealing-with-decimal-and-fractions%2F23377%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [September 20, 2022, 7:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdealing-with-decimal-and-fractions%2F23377 "2022-09-20T07:01:36Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![amkorba](https://avatars.discourse-cdn.com/v4/letter/a/71e660/32.png) [@amkorba](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Famkorba)%5C
**Post date:** [September 20, 2022, 7:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdealing-with-decimal-and-fractions%2F23377%2F1 "2022-09-20T07:01:36Z")

</div>

Hi Team,

I’m currently dealing with dataset which contains decimal and fraction and my task have big impact because of those numbers.

When I’m trying Bert based tokenizer, its tokenizing “1-1/2” as [1,-,1,/,2] but I want it as single token i.e [“1-1/2”]  
Something similar happening with decimals too.

Please suggest the possible solutions to tokenize it properly.

Thanks  
Ashish

---

<div class="post-metadata">

**Author:** ![lianghsun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lianghsun/32/57774_2.png) [@lianghsun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flianghsun)%5C
**Post date:** [October 27, 2022, 8:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdealing-with-decimal-and-fractions%2F23377%2F2 "2022-10-27T08:56:58Z")

</div>

Hi @amkorba, you should try retrain a Bert based tokenizer and using customized `Split()`. Let’s step by step to demostrate it 🤗.

Step 1: Suppose you have a dataset (fraction.txt) likes below:

```auto
Hi 1-1/2 there
Lorem oweh 3/4

```

Step 2: Load pre-trained Bert tokenizer to HF Tokenizers

```auto
from tokenizers import Tokenizer
from tokenizers.pre_tokenizers import Split

tokenizer = Tokenizer.from_pretrained("bert-base-uncased")

```

Step 3: Define the patterns to split fraction

```auto
from tokenizers import Regex

pre_tokenizer = Split(
    pattern=Regex("( ([1-9]\d*\-) | ([1-9]\d*\/[1-9]\d*) | ([1-9]\d*\-[1-9]\d*\/[1-9]\d*) )"),
    behavior='isolated'
)
pre_tokenizer.pre_tokenize_str('Hi 1-1/2 there')
# > [('Hi', (0, 2)), (' 1-1/2 ', (2, 9)), ('there', (9, 14))] <-- it works!

```

Step 4: Re-train Bert tokenizer to learn fraction into your vocab!

```auto
from tokenizers.trainers import WordPieceTrainer
trainer = WordPieceTrainer(
    vocab_size=30522, special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"]
)
files = ['/path/2/fraction.txt']
tokenizer.train(files, trainer)

```

Step 5: Check the new trained tokenizer!

```auto
tokenizer.encode("Hi 1-1/2 there").tokens
# > ['[CLS]', 'hi', ' 1-1/2 ', 'ther', '##e', '[SEP]']

```