# Format of data during pre-training

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fformat-of-data-during-pre-training%2F1388%3E%5C
**Category:** 🤗Datasets\
**Created:** [October 6, 2020, 3:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fformat-of-data-during-pre-training%2F1388 "2020-10-06T15:23:15Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![shainaraza](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/shainaraza/32/2854_2.png) [@shainaraza](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fshainaraza)%5C
**Post date:** [October 6, 2020, 3:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fformat-of-data-during-pre-training%2F1388%2F1 "2020-10-06T15:23:15Z")

</div>

What should be the format of the data for pre-training? could it be any raw data (e.g., news articles) in my case and then after I fine-tune, then I need to define it for a specific task e.g., classification?

---

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvblagoje)%5C
**Post date:** [October 7, 2020, 1:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fformat-of-data-during-pre-training%2F1388%2F2 "2020-10-07T13:13:09Z")

</div>

It comes in many forms and usually needs processing/adaptation for model input. Look into transformers examples to start.

If you are asking about the datasets project specifically - it makes it even simpler to do the above. There are many prepared datasets ready to go. Read through documentation.