# DPO training data format
**URL:**
**Author:**  [@Cagatayd](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FCagatayd)%5C
**Post date:** [September 22, 2024, 12:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpo-training-data-format%2F107755%2F1 "2024-09-22T12:45:35Z")
In preparing a dataset for DPO (Direct Preference Optimization) training, should the “prompt” be repeated in the “chosen” and “rejected” columns?
I’ve come across some conflicting information regarding the proper formatting of the dataset for DPO training. Some sources suggest that the prompt should be included in both the “chosen” and “rejected” responses to provide full context, while others state that the prompt should be kept separate and not repeated in these columns.
Additionally, when working with multi-turn dialogue data, I’m unsure how to properly format the dataset. Should the “chosen” and “rejected” columns include the entire conversation history up to that point, or just the assistant’s most recent response following the latest user input?
Could someone clarify the correct approach for formatting the dataset? Should the “chosen” and “rejected” columns contain only the assistant’s responses following the prompt, or should they include the prompt as well? And how should I handle multi-turn dialogues in this context?
I also wonder how to prepare multi turn conversation data such as Anthropic/hh-rlhf for DPO
and
Should we add “chosen\_rating” and “rejected\_rating” into dataset ?
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [September 22, 2024, 8:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpo-training-data-format%2F107755%2F2 "2024-09-22T20:44:30Z")
I’m not at all familiar with LLM training, so I can’t help you directly, but this person usually trains and reads papers on the RP model a lot and disseminates information, so I think you can get good information by sending a mentions (@+grimjim) or opening a New Discussion on his repo and contacting him.
> **[grimjim (Jim Lai)](https://huggingface.co/grimjim)**
>
> Experimenting primarily with 7B-12B parameter text completion models. Not all models are intended for direct use, but aim for educational and/or merge purposes.
---
**Author:**  [@Cagatayd](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FCagatayd)%5C
**Post date:** [September 22, 2024, 9:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpo-training-data-format%2F107755%2F3 "2024-09-22T21:30:40Z")
Hi, thanks, gonna ask him…
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [September 22, 2024, 10:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpo-training-data-format%2F107755%2F4 "2024-09-22T22:06:14Z")
His activities are irregular, so you should be patient. Also, there are probably quite a few individuals and researchers on HF who are familiar with DPO that I don’t know, but I don’t think many of them frequent the forum.
It would be more reliable to find and follow DPO-related repos and articles and contact them directly.
This is a current flaw in the HF community function and has been pointed out in various ways in the recent and still ongoing request for opinions.
> **[@victor on Hugging Face: "🙋 Calling all Hugging Face users! We want to hear...](https://huggingface.co/posts/victor/964839563451127)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
---
**Author:**  [@Cagatayd](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FCagatayd)%5C
**Post date:** [September 23, 2024, 8:51am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpo-training-data-format%2F107755%2F5 "2024-09-23T08:51:13Z")
Thanks a lot for your help, I did some research on DPO-related repos and articles and asked yesterday, If I get an answer, I’ll share it with you.
I also wonder how to prepare multi turn conversation data such as Anthropic/hh-rlhf for DPO
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [September 23, 2024, 11:49am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpo-training-data-format%2F107755%2F6 "2024-09-23T11:49:51Z")
> I did some research on DPO-related repos and articles and asked yesterday
Oh, you had already acted.
> such as Anthropic/hh-rlhf for DPO
I think the best way to be sure is to ask Anthropic directly, but if he seems taciturn, you could try to find another training data author who seems sociable.
I’m basically just a guy playing with image generation AI and Python, and I’m only tinkering a bit with LLM for tag generation, so I’ll just watch LLM training from afar and taste it when it’s finished.
Well, I’m limited to giving directions.
---
**Author:**  [@Cagatayd](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FCagatayd)%5C
**Post date:** [September 23, 2024, 12:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpo-training-data-format%2F107755%2F8 "2024-09-23T12:42:17Z")
I was confused because some people added prompt in ‘chosen’ and ‘rejected’ on their notebooks.a
I think the best way is seperating ‘prompt’, ‘chosen’ and ‘rejected’, according to GitHub repo(1) of TRL, DPOtrainer, and hf docs(2) during training it add prompt already I’ll separate columns to avoid repetition.
1- [trl/trl/trainer/dpo\_trainer.py at main · huggingface/trl · GitHub](https://github.com/huggingface/trl/blob/main/trl/trainer/dpo_trainer.py)
2- [https://huggingface.co/docs/trl/main/en/dpo\_trainer](https://huggingface.co/docs/trl/main/en/dpo_trainer)
3-[Fine-tune a Mistral-7b model with Direct Preference Optimization | by Maxime Labonne | Towards Data Science](https://towardsdatascience.com/fine-tune-a-mistral-7b-model-with-direct-preference-optimization-708042745aac)
4- [How to fine-tune Llama3 using DPO on Brev 🤙](https://www.brev.dev/blog/how-to-finetune-llama3) etc…
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [September 23, 2024, 1:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpo-training-data-format%2F107755%2F9 "2024-09-23T13:32:58Z")
There is no reliable introductory procedure for generative AI-related technologies in general. Even if there is, the speed of evolution makes them obsolete within a few months.
That may be partly why, in HF, there are so many people who are basically doing things on their own.
Some must be following outdated know-how without realizing it. This case may be it.
There are also probably many raw data sets that have been placed on the assumption that they will be used after being processed by scripts.