# Adding atomic / indivisible tokens to BPE tokenizer

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadding-atomic-indivisible-tokens-to-bpe-tokenizer%2F161486%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [July 2, 2025, 7:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadding-atomic-indivisible-tokens-to-bpe-tokenizer%2F161486 "2025-07-02T07:55:53Z")\
**Posts on this page:** 1\
**Showing post:** 7

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [July 3, 2025, 10:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadding-atomic-indivisible-tokens-to-bpe-tokenizer%2F161486%2F7 "2025-07-03T10:09:28Z")

</div>

Hmm… Like this?

```py
#tokenizer.pre_tokenizer = AtomicUnitPreTokenizer(atomic_units)
tokenizer.pre_tokenizer = PreTokenizer.custom(AtomicUnitPreTokenizer(atomic_units))

```

> <https://github.com/huggingface/tokenizers/issues/269>
>
> Hello, 
> 
> I was trying to subclass \`PreTokenizer\` class from \`tokenizers.pre\_to…kenizers\` in order to create my own custom pre-tokenizer for splitting the text at word level, but this doesn't seem to work since I'm getting the following:
> 
> \`\`\`python
> TypeError: type 'PreTokenizer' is not an acceptable base type.
> \`\`\`
> 
> Any ideas how I can do this? Thanks in advance.

> <https://github.com/huggingface/tokenizers/issues/1634>
>
> Hey,
> 
> I want to train a Tokenizer that operates on a custom PreTokenizer. I tr…ied a mix of \[this documentation post\](https://huggingface.co/docs/tokenizers/pipeline) and \[this example\](https://github.com/huggingface/tokenizers/blob/b24a2fc1781d5da4e6ebcd3ecb5b91edffc0a05f/bindings/python/examples/custom\_components.py). My resulting code looks like this:
> 
> \`\`\`python
> class GlyLESPreTokenizer:
> def \_\_init\_\_(self, \*args, \*\*kwargs):
> pass
> 
> def \_\_new\_\_(cls, \*args, \*\*kwargs):
> return super().\_\_new\_\_(cls)
>     
> def glyles\_split(self, iupac: str):
> iuapc = iupac.strip().replace(" ", "")
> token = CommonTokenStream(GlyLESLexer(InputStream(data="{" + iupac + "}")))
> GlyLESParser(token).start()
> idx = 0
> output = \[\]
> for i in range(1, len(token.tokens) - 2):
> txt = str(token.tokens\[i\].text)
> output.append((txt, (idx, idx + len(txt))))
> idx += len(txt)
> return output
>         
> def pre\_tokenize\_str(self, input\_: str):
> return self.glyles\_split(input\_)
> 
> 
> iupac = "QuiNAlaAc(b1-4)GalNAcA(a1-4)GalOAc(a1-2)QuiNAlaAc"
> 
> \# This returns a list of 33 token
> GlyLESPreTokenizer().pre\_tokenize\_str(iupac)
> 
> \# This however only returns a list with one token that is the entire input string.
> pre\_tokenizers.PreTokenizer.custom(GlyLESPreTokenizer()).pre\_tokenize\_str(iupac)
> \`\`\`
> 
> The final idea is to use it in such setting:
> \`\`\`python
> tokenizer = Tokenizer(models.Model())
> tokenizer.normalizer = normalizers.Strip()
> tokenizer.pre\_tokenizer = pre\_tokenizers.PreTokenizer.custom(GlyLESPreTokenizer())
> \`\`\`
> 
> Can someone help me to understand how to use the \`pre\_tokenizers.PreTokenizer.custom\` method to inject a custom, python-written PreTokenizer into a Tokenizer? Unfortunately, it is far beyond the scope of the project to convert the logic from \[GlyLES\](https://github.com/kalininalab/GlyLES) to RUST, so it has to be a Python PreTokenizer-class that is somehow injected into the Tokenizer.
> 
> Thank you for any help, comment, or feedback in advance.
> Roman

> [@How to add additional custom pre-tokenization processing?](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-add-additional-custom-pre-tokenization-processing%2F1637)%3A
>
> I would like to add a few custom functions for pre-tokenization. For example, I would like to split numerical text from any non-numerical test. Eg ‘1000mg’ would become [‘1000’, ‘mg’]. I am trying to figure out the proper way to do this for the python binding; I think it may be a bit tricky since its a binding for the original rust version. I am looking at the pretokenizer function /huggingface/tokenizers/blob/2ccd16bf5c3dd97759d7bdf5229e2feeba314b4a/bindings/python/py\_src/tokenizers/pre\_to…

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadding-atomic-indivisible-tokens-to-bpe-tokenizer%2F161486)._