# Tokenizer: what function removes spaces between '\<' and '\>'?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-what-function-removes-spaces-between-and%2F130257%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [December 9, 2024, 7:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-what-function-removes-spaces-between-and%2F130257 "2024-12-09T19:46:29Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![idruker](https://avatars.discourse-cdn.com/v4/letter/i/3bc359/32.png) [@idruker](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fidruker)%5C
**Post date:** [December 9, 2024, 7:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-what-function-removes-spaces-between-and%2F130257%2F1 "2024-12-09T19:46:29Z")

</div>

Here is an detokenized sequence:

print(tokenizer.decode(generated\_ids[0][input\_ids.shape[-1]:], skip\_special\_tokens=False, clean\_up\_tokenization\_space=False))

\<tool\_call\>  
{“arguments”: {“symbol”: “AAPL”}, “name”: “get\_stock\_fundamentals”}  
\</tool\_call\>\<|im\_end|\>

As you see, \</tool\_call\> is a construct without spaces. The original sequence of token ids is  
700 6462 28730 2845 28767 which is  
“\</” “tool” “\_” “call” “\>”

What Transformers function implements the logic of removal of the spaces? How does it know that ‘tool’, ‘\_’ and ‘call’ are part of one keyword?

Would appreciate your guidance.