# Tokenizer method inference
**URL:**
**Author:**  [@jinoooooooooo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjinoooooooooo)%5C
**Post date:** [November 1, 2024, 6:31am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-method-inference%2F114974%2F1 "2024-11-01T06:31:17Z")
This is a beginner question but
How does huggingface infer what type of method to use for tokenizing the sentence? Can you point me where the implementation could be?
Thanks
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [November 2, 2024, 1:39am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-method-inference%2F114974%2F2 "2024-11-02T01:39:18Z")
Maybe simply written in a json file in the same folder.
> **[tokenizer\_config.json · unsloth/Llama-3.2-1B-Instruct at main](https://huggingface.co/unsloth/Llama-3.2-1B-Instruct/blob/main/tokenizer_config.json)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
---
**Author:**  [@jinoooooooooo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjinoooooooooo)%5C
**Post date:** [November 2, 2024, 6:14am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-method-inference%2F114974%2F3 "2024-11-02T06:14:27Z")
not really, i couldnt find it
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [November 2, 2024, 6:32am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-method-inference%2F114974%2F4 "2024-11-02T06:32:41Z")
Oh, you’re looking for code. The following is called based on the YAML set in the model repo and the json as above. in Llama’s case, this is it.
>