# Difference between tokenizer and convert\_tokens\_to\_ids

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdifference-between-tokenizer-and-convert-tokens-to-ids%2F86361%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [May 12, 2024, 5:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdifference-between-tokenizer-and-convert-tokens-to-ids%2F86361 "2024-05-12T17:57:11Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![mcallara](https://avatars.discourse-cdn.com/v4/letter/m/f9ae1b/32.png) [@mcallara](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmcallara)%5C
**Post date:** [May 12, 2024, 5:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdifference-between-tokenizer-and-convert-tokens-to-ids%2F86361%2F1 "2024-05-12T17:57:11Z")

</div>

I was trying to convert tokens that contain spaces into ids and realized that I don’t get the same result if I use convert\_tokens\_to\_ids. Shouldn’t this map to the same ids?  
Thank you for your help.

```auto
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
            "meta-llama/Meta-Llama-3-8B",
            use_fast=False,
        )

# This returns None for both
tokenizer.convert_tokens_to_ids([" "," ,"])
# This returns the correct ids (220 and 1174)
tokenizer([" "," ,"],add_special_tokens=False).input_ids
# To get the right tokens I need to replace spaces with Ġ, shouldn't this be handle by the convert_tokens_to_ids method?
tokenizer.convert_tokens_to_ids(["Ġ","Ġ,"])

```