# Can we use a random state Bert model in BertGeneration?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-we-use-a-random-state-bert-model-in-bertgeneration%2F43213%3E%5C
**Category:** 🤗Transformers\
**Created:** [June 14, 2023, 11:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-we-use-a-random-state-bert-model-in-bertgeneration%2F43213 "2023-06-14T11:04:52Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![AliMGH](https://avatars.discourse-cdn.com/v4/letter/a/2bfe46/32.png) [@AliMGH](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAliMGH)%5C
**Post date:** [June 14, 2023, 11:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-we-use-a-random-state-bert-model-in-bertgeneration%2F43213%2F1 "2023-06-14T11:04:52Z")

</div>

I want to create an `EncoderDecoderModel` for a translation task using a `Bert2Bert` configuration, where the encoder model is pre-trained and frozen, and the decoder model is randomly initialized. In the `BertGeneration` [document](https://huggingface.co/docs/transformers/model_doc/bert-generation), it says:

> _We developed a Transformer-based sequence-to-sequence model that is compatible with publicly available pre-trained BERT, GPT-2 and RoBERTa checkpoints and conducted an extensive empirical study on the utility of initializing our model, both encoder and decoder, with these checkpoints._

Can I use a new randomly initialized model in the `BertGenerationDecoder`? or it is better to use `BertGeneration` with pre-trained models?

```auto
model_config = BertConfig(
    vocab_size=tokenizer.vocab_size,
    hidden_size=hidden_size,
    add_cross_attention=True,
    is_decoder=True,
    bos_token_id=tokenizer.cls_token_id,
    eos_token_id=tokenizer.sep_token_id,
)
model = BertGenerationDecoder(config=model_config)

```