# Emojis poisoning tokenizer

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Femojis-poisoning-tokenizer%2F92479%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [June 17, 2024, 2:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Femojis-poisoning-tokenizer%2F92479 "2024-06-17T02:09:03Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dts1](https://avatars.discourse-cdn.com/v4/letter/d/a587f6/32.png) [@Dts1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDts1)%5C
**Post date:** [June 17, 2024, 2:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Femojis-poisoning-tokenizer%2F92479%2F1 "2024-06-17T02:09:03Z")

</div>

I have a problem with emojis “poisoning” tokenizers and I wonder if there is an existing solution to deal with this problem. So far, I was not able to find it by doing different web searches. I have some ideas how to solve it by myself but I’d like to know if somebody already dealt with it.

The problem is the following: is an emoji is not separated from a word, then the whole word is marked as UNK token. Example:

from transformers import AutoTokenizer  
tokenizer = AutoTokenizer.from\_pretrained(“sentence-transformers/distiluse-base-multilingual-cased-v1”)  
text=“sky”  
print(tokenizer(text).input\_ids)  
text=“sky🙂”  
print(tokenizer(text).input\_ids)  
text=“😒sky”  
print(tokenizer(text).input\_ids)

Output:  
[101, 62368, 102]  
[101, 100, 102]  
[101, 100, 102]