# Question about llama fine tuning dataset token string

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestion-about-llama-fine-tuning-dataset-token-string%2F155584%3E%5C
**Category:** Beginners\
**Created:** [May 16, 2025, 9:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestion-about-llama-fine-tuning-dataset-token-string%2F155584 "2025-05-16T21:00:22Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [May 17, 2025, 6:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestion-about-llama-fine-tuning-dataset-token-string%2F155584%2F2 "2025-05-17T06:38:50Z")

</div>

Not limited to \<\>, special characters that have no meaning in themselves should not be passed unless there is a specific intention to do so. Generally, it is better to have as little noise as possible in data. For chat or writing models, it is better not to pass non-conversational character strings. (This does not apply when \<\> has a specific meaning, such as the emoticon \>\_\<.)

However, when training a model, Chat Templates or special tokens are important, but their assignment should be automated to some extent by the Tokenizer. (This is the default behavior unless explicitly specified.)

> [@SFT Trainer and chat templates](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsft-trainer-and-chat-templates%2F147205)%3A
>
> Hello I’m implementing a framework for fine-tuning various LLMs using the TRL library’s SFTTrainer. I have a question about how chat templates work: When using SFTTrainer with datasets in the standard formats (with “messages” array or “prompt”/“completion” fields), does the trainer automatically apply the tokenizer’s chat\_template? The documentation suggests it does. For models whose tokenizers don’t have a chat\_template attribute set (or it’s empty), what template does SFTTrainer apply by def…

> **[Templates](https://huggingface.co/docs/transformers/main/en/chat_templating)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

> **[Train a Llama model from scratch](https://huggingface.co/blog/nroggendorff/train-with-llama-architecture)**
>
> A Blog post by Noa Roggendorff on Hugging Face

> **[Introduction - Hugging Face LLM Course](https://huggingface.co/learn/llm-course/chapter1/1)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestion-about-llama-fine-tuning-dataset-token-string%2F155584)._