# Pretrain encoder of tf T5 model

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-encoder-of-tf-t5-model%2F1627%3E%5C
**Category:** Intermediate\
**Created:** [October 19, 2020, 2:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-encoder-of-tf-t5-model%2F1627 "2020-10-19T14:15:31Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![dhara](https://avatars.discourse-cdn.com/v4/letter/d/ba9def/32.png) [@dhara](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdhara)%5C
**Post date:** [October 19, 2020, 2:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-encoder-of-tf-t5-model%2F1627%2F1 "2020-10-19T14:15:31Z")

</div>

Hi, for sequence to sequence task in tensorflow, I see that I can use TFT5 model. However, before sequence to sequence training, I need to perform masked language model pretraining on the encoder and initialise the weights of the encoder and decoder with the same weights that I obtain by masked language model pretraining. Is there a way to do this in the tensorflow version of T5?  
I could do that with the EncoderDecoder, but it is only supported in PyTorch and not tensorflow.

Please help.