# Question about llama fine tuning dataset token string

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestion-about-llama-fine-tuning-dataset-token-string%2F155584%3E%5C
**Category:** Beginners\
**Created:** [May 16, 2025, 9:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestion-about-llama-fine-tuning-dataset-token-string%2F155584 "2025-05-16T21:00:22Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![kbuwel](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kbuwel/32/46076_2.png) [@kbuwel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkbuwel)%5C
**Post date:** [May 16, 2025, 9:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestion-about-llama-fine-tuning-dataset-token-string%2F155584%2F1 "2025-05-16T21:00:22Z")

</div>

Sorry. No matter how many times I press the reply and reply buttons with the Enter and Spacebar, the editing window where I can write a post does not appear, so I had no choice but to open a new topic.  
In my previous post, I uploaded the code and asked how to fine-tune it well.  
And I modified it according to the answer.  
Now, I am experimenting with how to structure the dataset to get good results.  
In my previous post, I showed an example.  
I received an answer that special characters should not be included when fine-tuning.  
Here, I have a little question.  
gpt3 recommended ~~and~~ to me.  
It said to use ~~when starting the body and~~ when the body ends.  
However, in my previous post, it said not to use \<\>.  
According to the explanation in gpt3, you need to input ~~and~~ so that the model can know the beginning and end of the body of the novel data.  
Should I not use ~~and~~?  
Can I just input the body without using this string?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [May 17, 2025, 6:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestion-about-llama-fine-tuning-dataset-token-string%2F155584%2F2 "2025-05-17T06:38:50Z")

</div>

Not limited to \<\>, special characters that have no meaning in themselves should not be passed unless there is a specific intention to do so. Generally, it is better to have as little noise as possible in data. For chat or writing models, it is better not to pass non-conversational character strings. (This does not apply when \<\> has a specific meaning, such as the emoticon \>\_\<.)

However, when training a model, Chat Templates or special tokens are important, but their assignment should be automated to some extent by the Tokenizer. (This is the default behavior unless explicitly specified.)

> [@SFT Trainer and chat templates](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsft-trainer-and-chat-templates%2F147205)%3A
>
> Hello I’m implementing a framework for fine-tuning various LLMs using the TRL library’s SFTTrainer. I have a question about how chat templates work: When using SFTTrainer with datasets in the standard formats (with “messages” array or “prompt”/“completion” fields), does the trainer automatically apply the tokenizer’s chat\_template? The documentation suggests it does. For models whose tokenizers don’t have a chat\_template attribute set (or it’s empty), what template does SFTTrainer apply by def…

> **[Templates](https://huggingface.co/docs/transformers/main/en/chat_templating)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

> **[Train a Llama model from scratch](https://huggingface.co/blog/nroggendorff/train-with-llama-architecture)**
>
> A Blog post by Noa Roggendorff on Hugging Face

> **[Introduction - Hugging Face LLM Course](https://huggingface.co/learn/llm-course/chapter1/1)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.