# PreTrain BART on The Pile
**URL:**
**Author:**  [@LouisCastricato](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLouisCastricato)%5C
**Post date:** [June 24, 2021, 6:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-bart-on-the-pile%2F7286%2F1 "2021-06-24T18:42:21Z")
Description: We will pretrain a large BART model on The Pile, and measure a performance increase downstream. Potentially we could also add rotary embeddings?
Model: BART (1b+)
Dataset: The Pile
Training scripts: Training scripts will be written as part of the project. Data processing scripts can be taken from GPT-J 6b
Expected result: An adaptable JAX pipeline for training seq2seq models like BART on the pile.
---
**Author:**  [@paws](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpaws)%5C
**Post date:** [June 25, 2021, 1:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-bart-on-the-pile%2F7286%2F2 "2021-06-25T13:00:54Z")
Im Interested in this project, Any reasons why you’d aim for BART and not any model with some architectural improvements?
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 25, 2021, 5:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-bart-on-the-pile%2F7286%2F3 "2021-06-25T17:22:36Z")
Great project! I’d also be interested in why one should use BART. Maybe T5 is also an option? We’ll have an official pretraining script merged for T5 very soon → see: [[Flax] Add T5 pretraining script by patrickvonplaten · Pull Request #12355 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/pull/12355)
---
**Author:**  [@LouisCastricato](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLouisCastricato)%5C
**Post date:** [June 25, 2021, 6:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-bart-on-the-pile%2F7286%2F4 "2021-06-25T18:14:50Z")
To be honest it’s just because I am familiar with BART 🤷♂️ I haven’t really used T5 in practice yet. A lot of a stack I built last year is still using BART.
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [June 26, 2021, 7:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-bart-on-the-pile%2F7286%2F5 "2021-06-26T07:11:38Z")
Cool idea!
IMO one important consideration would be the oupt sequence length. AFAIU for BART’s denoiseing objective the output sequence length is same as the input length,where as for T5 it’s quite small, which would lead to faster training.
adding rotary embeddings also seems like a good idea!
Also what do you think about [deep encoder-shallow decoder](https://arxiv.org/abs/2006.10369), looking at the ByT5 paper it seems it’s worth exploring.
---
**Author:**  [@paws](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpaws)%5C
**Post date:** [June 26, 2021, 10:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-bart-on-the-pile%2F7286%2F6 "2021-06-26T10:48:56Z")
The BART model’s objective seems to help better than T5 though, based on the results for text generation
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [June 28, 2021, 5:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-bart-on-the-pile%2F7286%2F7 "2021-06-28T17:19:24Z")
Let’s officially define this project 🙂
Putting everybody in the official sheet [here](https://docs.google.com/spreadsheets/d/1GpHebL7qrwJOc9olTpIPgjf8vOS0jNb6zR_B8x_Jtik/edit?usp=sharing). More people can still join! Leave a comment here or on the sheet if you want to change something.
---
**Author:**  [@LouisCastricato](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLouisCastricato)%5C
**Post date:** [June 28, 2021, 5:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-bart-on-the-pile%2F7286%2F8 "2021-06-28T17:24:39Z")
@THEODOROS was interested in doing cross project collaboration, he mentioned wanting to use the BART+rotary implementation we end up with.
---
**Author:**  [@morgan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmorgan)%5C
**Post date:** [June 29, 2021, 10:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-bart-on-the-pile%2F7286%2F9 "2021-06-29T10:40:23Z")
This is a cool idea, would love to help out if I can! Adding myself to the google sheet (if thats cool)
---
**Author:**  [@LouisCastricato](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLouisCastricato)%5C
**Post date:** [June 30, 2021, 3:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-bart-on-the-pile%2F7286%2F10 "2021-06-30T03:27:18Z")
Join on the discord @morgan
---
**Author:**  [@paws](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpaws)%5C
**Post date:** [June 30, 2021, 7:35am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-bart-on-the-pile%2F7286%2F11 "2021-06-30T07:35:38Z")
@valhalla @patrickvonplaten We wanted to make some architectural improvements in the BART model, some of it related to using deberta’s tokenizer and also adding rotary embeddings, how would we use this model in HF after that?
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [June 30, 2021, 9:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-bart-on-the-pile%2F7286%2F12 "2021-06-30T09:05:36Z")
Hi @paws
Feel free to make any improvements you want. You could see how `FlaxBart` is implemented and try to keep the same API (i.e ` __call__ `, `save_pretrained` and `from_pretrained`), that way the model will be compatible with HF API. And for now, you could create new repo, we could always modify the code later to make it compatible with HF API.
---
**Author:**  [@morgan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmorgan)%5C
**Post date:** [June 30, 2021, 10:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-bart-on-the-pile%2F7286%2F13 "2021-06-30T10:29:44Z")
Thanks @LouisCastricato , where is the discord link? Or do you mean the HF slack? Haven’t received an invite for that yet…
---
**Author:**  [@LouisCastricato](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLouisCastricato)%5C
**Post date:** [June 30, 2021, 3:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-bart-on-the-pile%2F7286%2F14 "2021-06-30T15:16:06Z")
You should talk to someone working at HF to get the invite. I do not think I am allowed to send or post the link. @valhalla do you mind helping morgan?
---
**Author:**  [@mattbui](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmattbui)%5C
**Post date:** [June 30, 2021, 4:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-bart-on-the-pile%2F7286%2F15 "2021-06-30T16:09:03Z")
@LouisCastricato Hi, I have experience with using BART and T5 for summarization before so I’m interested in this project, hope I can join.
---
**Author:**  [@LouisCastricato](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLouisCastricato)%5C
**Post date:** [June 30, 2021, 4:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-bart-on-the-pile%2F7286%2F16 "2021-06-30T16:38:20Z")
Of course! Join the discord as well!
---
**Author:**  [@morgan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmorgan)%5C
**Post date:** [June 30, 2021, 4:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-bart-on-the-pile%2F7286%2F17 "2021-06-30T16:57:41Z")
In, thanks!
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [July 1, 2021, 10:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-bart-on-the-pile%2F7286%2F18 "2021-07-01T10:17:00Z")
Added you as well @mattbui
---
**Author:**  [@tylerjthomas9](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftylerjthomas9)%5C
**Post date:** [July 1, 2021, 7:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-bart-on-the-pile%2F7286%2F19 "2021-07-01T19:34:06Z")
I have some experience with using BART for summarization on financial text. Very interested in learning more about pre-training seq2seq models + using TPU vms. I would love to join the project if there is still room on the team.
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [July 1, 2021, 11:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-bart-on-the-pile%2F7286%2F20 "2021-07-01T23:50:24Z")
added you 🙂