# Pre-training googlebyt5small

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-googlebyt5small%2F25028%3E%5C
**Category:** 🤗Transformers\
**Created:** [October 26, 2022, 3:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-googlebyt5small%2F25028 "2022-10-26T15:52:57Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![antoine2323231](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@antoine2323231](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fantoine2323231)%5C
**Post date:** [October 26, 2022, 3:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-googlebyt5small%2F25028%2F1 "2022-10-26T15:52:57Z")

</div>

Is it possible to use the run\_t5\_mlm\_flax.py script (link below) in order to pre-train the googlebyt5small and google/t5-small-lm-adapt models. Or there is a different script in order to do so?

> **[transformers/examples/flax/language-modeling at main · huggingface/transformers](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling)**
>
> main/examples/flax/language-modeling

Thanks in advance…