# Pretrain T5 for Arabic
**URL:**
**Author:**  [@salti](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsalti)%5C
**Post date:** [July 2, 2021, 9:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-arabic%2F7939%2F1 "2021-07-02T21:07:20Z")
# T5 for Arabic
Currently there is a fair amount of Encoder-only and Decoder-only models for Arabic (AraBERT, AraElectra, AraGPT2, etc.), but there aren’t any seq2seq models. The goal of this project is to pretrain a T5 language model for the Arabic language.
## Model
A randomly initialized T5 model. I’m not sure yet what the biggest model size that can be trained during a one week period however. Ideally, I think we should train the biggest variant possible, given that bigger models converge faster.
## Datasets
- The Arabic subset of the [mC4 dataset](https://huggingface.co/datasets/mc4)
- The Arabic subset of the [Oscar dataset](https://huggingface.co/datasets/oscar)
The Arabic subset of mC4 should be more than enough though. It contains 57 Billion tokens (according to the [mT5 paper](https://arxiv.org/abs/2010.11934)) and the English T5 model was trained on 2^{35} (about 34 Billion) tokens ([T5 paper](https://arxiv.org/abs/1910.10683)).
## Training scripts
We can use the recently released [T5 pretraining script](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_t5_mlm_flax.py)
## Challenges
I have never worked with datasets this huge.
## Desired outcomes
- An Arabic T5 model with reasonable results when fine-tuned for summarization, question generation, question answering and so on.
- Learning some JAX and having fun 😄
## Team members:
- ManarAli
- Onyx
- salti
---
**Author:**  [@ManarAli](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FManarAli)%5C
**Post date:** [July 3, 2021, 11:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-arabic%2F7939%2F2 "2021-07-03T23:11:37Z")
I am very interested in this project.
Are you still looking for partners?
---
**Author:**  [@salti](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsalti)%5C
**Post date:** [July 4, 2021, 1:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-arabic%2F7939%2F3 "2021-07-04T13:51:26Z")
Yes! I was actually the only member so far, so I definitely could use some help.
---
**Author:**  [@Onyx](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FOnyx)%5C
**Post date:** [July 5, 2021, 8:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-arabic%2F7939%2F4 "2021-07-05T20:06:06Z")
I’m also interested in joining.
---
**Author:**  [@salti](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsalti)%5C
**Post date:** [July 5, 2021, 9:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-arabic%2F7939%2F5 "2021-07-05T21:15:43Z")
Great! I’ll update the post with the current team members.
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [July 7, 2021, 10:35am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-arabic%2F7939%2F6 "2021-07-07T10:35:11Z")
Let’s create it 🙂
---
**Author:**  [@ghofrani](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fghofrani)%5C
**Post date:** [July 7, 2021, 12:40pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-arabic%2F7939%2F7 "2021-07-07T12:40:37Z")
I’m so interested in joining.
I Just want to use this way for Persian/Farsi Language.
Are you still looking for partners?
---
**Author:**  [@salti](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsalti)%5C
**Post date:** [July 7, 2021, 4:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-arabic%2F7939%2F8 "2021-07-07T16:48:23Z")
I’m not sure this project will be very useful for Persian. It’s much better to to pretrain a new model from scratch for Persian directly. The [mc4](https://huggingface.co/datasets/mc4) dataset also has a Persian subset which you can use for pretraining.
---
**Author:**  [@AFA](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAFA)%5C
**Post date:** [August 23, 2021, 11:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-arabic%2F7939%2F9 "2021-08-23T23:02:39Z")
Does mt5 support Arabic?
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [August 24, 2021, 7:12am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-arabic%2F7939%2F10 "2021-08-24T07:12:20Z")
Yes, looking at page 12 of mT5’s [original paper](https://arxiv.org/pdf/2010.11934.pdf), Arabic is indeed included in the pre-training corpus (57 billion tokens, 53 million pages, or 1.66 % of the total pre-training data).
---
**Author:**  [@AFA](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAFA)%5C
**Post date:** [August 24, 2021, 11:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-arabic%2F7939%2F11 "2021-08-24T11:53:39Z")
OK. Is that meaning; I can finetune mt5 to my Arabic dataset on task-oriented dialogue system tasks?
---
**Author:**  [@salti](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsalti)%5C
**Post date:** [August 24, 2021, 12:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-arabic%2F7939%2F12 "2021-08-24T12:25:15Z")
Yes. You can fine-tune T5 models on virtually any task if you formulate it as a text-to-text task.
---
**Author:**  [@AFA](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAFA)%5C
**Post date:** [August 24, 2021, 1:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-arabic%2F7939%2F13 "2021-08-24T13:53:35Z")
OK. thank you
---
**Author:**  [@Mohamed-Aziz](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMohamed-Aziz)%5C
**Post date:** [September 29, 2021, 9:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-arabic%2F7939%2F14 "2021-09-29T21:20:33Z")
any thing i can help with you guys ?
---
**Author:**  [@Arij](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FArij)%5C
**Post date:** [October 28, 2021, 10:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-arabic%2F7939%2F15 "2021-10-28T10:13:29Z")
Is the team completed, or still need members?
---
**Author:**  [@hkzak](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhkzak)%5C
**Post date:** [June 2, 2023, 11:03am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-arabic%2F7939%2F16 "2023-06-02T11:03:07Z")
Any updates on this ? this is the best time to make it work. Lets revive this if not yet finished
---
**Author:**  [@salti](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsalti)%5C
**Post date:** [June 10, 2023, 8:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-arabic%2F7939%2F17 "2023-06-10T20:42:32Z")
Hello Zakarya, the trained model was published on the hub right after the event concluded two years ago.
You can find it [here](https://huggingface.co/flax-community/arabic-t5-small), I also fine-tuned the model for [question paraphrasing](https://huggingface.co/salti/arabic-t5-small-question-paraphrasing).
The results are not bad but also not that great, it’s because we couldn’t make the training work until the last day or so of the event.
Let me know if you have interest in re-training the model or any other ideas.
---
**Author:**  [@hkzak](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhkzak)%5C
**Post date:** [June 11, 2023, 3:37am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-arabic%2F7939%2F18 "2023-06-11T03:37:46Z")
Hi Salti,
That’s really good, I would love to play with the model and further train or finetune it.
If you have LinkedIn we can connect there. Here is the link to my profile: httpe://[www.linkedin.com/in/zakarya-alsalahi](http://www.linkedin.com/in/zakarya-alsalahi)
Regards,
Zak