# Data collator for training bart from scratch

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdata-collator-for-training-bart-from-scratch%2F2047%3E%5C
**Category:** Beginners\
**Created:** [November 13, 2020, 9:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdata-collator-for-training-bart-from-scratch%2F2047 "2020-11-13T09:13:23Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![zuujhyt](https://avatars.discourse-cdn.com/v4/letter/z/f17d59/32.png) [@zuujhyt](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzuujhyt)%5C
**Post date:** [November 13, 2020, 9:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdata-collator-for-training-bart-from-scratch%2F2047%2F1 "2020-11-13T09:13:23Z")

</div>

Hello,  
I would like to train bart from scratch.  
It seems the official example script is not available yet (if any, please tell me!).  
So I try to have one by modifying the example scripts `run_mlm.py` and `run_clm.py`.  
And not sure how to set the data collator part for bart.

In `run_mlm.py` , DataCollatorForLanguageModeling is used:

> <https://github.com/huggingface/transformers/blob/0c9bae09340dd8c6fdf6aa2ea5637e956efe0f7c/examples/language-modeling/run_mlm.py#L342>

In `run_clm.py` , default\_data\_collator is used:

> <https://github.com/huggingface/transformers/blob/0c9bae09340dd8c6fdf6aa2ea5637e956efe0f7c/examples/language-modeling/run_clm.py#L310>

Can someone give some advice about how to set the data collator for bart?  
Thanks.

---

<div class="post-metadata">

**Author:** ![dhruvramani](https://avatars.discourse-cdn.com/v4/letter/d/258eb7/32.png) [@dhruvramani](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdhruvramani)%5C
**Post date:** [December 6, 2020, 1:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdata-collator-for-training-bart-from-scratch%2F2047%2F3 "2020-12-06T13:19:58Z")

</div>

[@zuujhyt](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzuujhyt) Maybe try creating different `Dataset` classes using different `DataCollator`s. You could then use PyTorch Lightning to create a `Dataloader` w/ multiple `Dataset` classes like [this](https://pytorch-lightning.readthedocs.io/en/latest/multiple_loaders.html), and train Bart on that. I’ll try doing this.

Let me know if you have any updates on this.