# Warning when loading T5 encoders
**URL:**
**Author:**  [@jhuang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjhuang)%5C
**Post date:** [April 8, 2021, 11:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwarning-when-loading-t5-encoders%2F5377%2F1 "2021-04-08T23:00:20Z")
I am trying to load t5-large (same for t5-small and t5-base) encoder as a feature extractor. I use the following command
```auto
from transformers import T5EncoderModel
M=T5EncoderModel.from_pretrained('t5-large')
```
But I get warning msgs like this:
```auto
Some weights of the model checkpoint at t5-large were not used when initializing T5EncoderModel:
['decoder.block.0.layer.0.SelfAttention.q.weight', ....,'decoder.final_layer_norm.weight']
- This IS expected if you are initializing T5EncoderModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing T5EncoderModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of T5EncoderModel were not initialized from the model checkpoint
at t5-large and are newly initialized: ['encoder.embed_tokens.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```
I can understand that the warning on decoder is because I only load encoder part. However, I am a bit worried about [‘encoder.embed\_tokens.weight’]. Is the encoder (as it is) reliable for feature extraction?
btw, I also find that the parameter value of ‘encoder.embed\_tokens.weight’ equals the look-up embedding layer, ‘shared.weight’.
```auto
torch.equal(M.state_dict()['encoder.embed_tokens.weight'], M.state_dict()['shared.weight']) == True
```
Anyone help me understand the issue?
---
**Author:**  [@arunwzd](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Farunwzd)%5C
**Post date:** [May 8, 2022, 6:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwarning-when-loading-t5-encoders%2F5377%2F2 "2022-05-08T06:58:27Z")
any update on this?
---
**Author:**  [@arunwzd](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Farunwzd)%5C
**Post date:** [June 8, 2022, 3:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwarning-when-loading-t5-encoders%2F5377%2F3 "2022-06-08T15:39:12Z")
@sgugger can you help me with this?
---
**Author:**  [@Reza8848](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FReza8848)%5C
**Post date:** [May 15, 2023, 3:45am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwarning-when-loading-t5-encoders%2F5377%2F4 "2023-05-15T03:45:18Z")
It is a normal phenomenon: `'encoder.embed_tokens.weight'` will be initialized randomly (using `self.shared`) instead of pre-trained weights.
This warning was ignored in the newest version of the `transformer`.
And of course, it is harmless; you can feel free to ignore this warning.
Please refer to this [comment](https://github.com/huggingface/transformers/issues/3553#issuecomment-624306027) for more details.