# Pre-train DistilmByT5Neo
**URL:**
**Author:**  [@wolosonovich](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwolosonovich)%5C
**Post date:** [June 29, 2021, 2:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-distilmbyt5neo%2F7586%2F1 "2021-06-29T02:53:28Z")
**Pre-train DistilmByT5Neo**
Let’s combine ByT5, mT5, GPT-Neo and DistilBert!
**Model**
For initial pre-training, a randomly initialized ByT5 model which we distil after pre-training is completed.
**Datasets**
[c4multilingual](https://www.tensorflow.org/datasets/catalog/c4#c4multilingual)
[The Pile](https://the-eye.eu/public/AI/pile/)
**Training Scripts**
Training scripts will be created as part of the project
**Expected Result**
A Distilled model that combines the power of T5 and GPT-Neo while removing the need for tokenization.
UPDATE: We could just go for it and implement this with rotary embeddings as well.
---
**Author:**  [@wolosonovich](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwolosonovich)%5C
**Post date:** [June 29, 2021, 7:12am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-distilmbyt5neo%2F7586%2F2 "2021-06-29T07:12:46Z")
We could kick this up a notch further with rotary embeddings…DistilmRoByT5Neo?
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 30, 2021, 7:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-distilmbyt5neo%2F7586%2F3 "2021-06-30T19:36:34Z")
Finalizing since another team member will join 🙂 @wolosonovich it would be great if you could post the hub name of your team member here once you have it
---
**Author:**  [@wolosonovich](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwolosonovich)%5C
**Post date:** [June 30, 2021, 7:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-distilmbyt5neo%2F7586%2F4 "2021-06-30T19:44:03Z")
I will do that, thank you very much @patrickvonplaten !
---
**Author:**  [@wolosonovich](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwolosonovich)%5C
**Post date:** [July 1, 2021, 6:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-distilmbyt5neo%2F7586%2F5 "2021-07-01T18:27:23Z")
@patrickvonplaten can you add @vmazelis to the spreadsheet. he is a part of our team as well for this project.
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [July 1, 2021, 10:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-distilmbyt5neo%2F7586%2F6 "2021-07-01T22:41:20Z")
should be done 🙂
---
**Author:**  [@wolosonovich](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwolosonovich)%5C
**Post date:** [July 2, 2021, 6:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-distilmbyt5neo%2F7586%2F7 "2021-07-02T18:30:40Z")
@patrickvonplaten can you update the spreadsheet and replace Brett with @bneb10 when you have a chance? thanks so much!
---
**Author:**  [@wolosonovich](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwolosonovich)%5C
**Post date:** [July 2, 2021, 6:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-distilmbyt5neo%2F7586%2F8 "2021-07-02T18:58:28Z")
link to our discord server [Flax-HuggingFace-Community-Week](https://discord.gg/CJU6U5xpaZ)