# Pretrain T5 from scratch in Chinese

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-from-scratch-in-chinese%2F8124%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [July 7, 2021, 7:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-from-scratch-in-chinese%2F8124 "2021-07-07T07:53:01Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![chewkokwah](https://avatars.discourse-cdn.com/v4/letter/c/919ad9/32.png) [@chewkokwah](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fchewkokwah)%5C
**Post date:** [July 7, 2021, 7:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-from-scratch-in-chinese%2F8124%2F1 "2021-07-07T07:53:01Z")

</div>

**Pretrain T5 for Chinese**  
Currently there is No T5 pretrained model for Chinese language on the Huggingface hub. The goal is to train a T5-base model in Chinese and finetune it on Chinese QA dataset CMRC2018 to see its performance compare to other non-T5 model.

**Model**  
A randomly initialized T5 model

**Datasets**  
CLUECorpus2020  
OSCAR  
mC4  
CMRC2018 ([CMRC 2018](https://ymcui.com/cmrc2018/))

**Available training scripts**  
Will be using standard T5 training script from Huggingface

**(Optional) Desired project outcome**

A ChineseT5 model that is competitive with other similar size/speed non-T5 model (Roberta, GPTs etc） on Chinese QA dataset

**(Optional) Challenges**

Chinese character is non-spaced, might need special treatment compare to spaced language like English.

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [July 7, 2021, 10:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-from-scratch-in-chinese%2F8124%2F2 "2021-07-07T10:30:15Z")

</div>

sounds good - let’s define it!