# Use external embeddings

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fuse-external-embeddings%2F20316%3E%5C
**Category:** 🤗Transformers\
**Created:** [July 13, 2022, 9:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fuse-external-embeddings%2F20316 "2022-07-13T09:07:32Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![adir](https://avatars.discourse-cdn.com/v4/letter/a/e47c2d/32.png) [@adir](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fadir)%5C
**Post date:** [July 13, 2022, 9:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fuse-external-embeddings%2F20316%2F1 "2022-07-13T09:07:32Z")

</div>

Hi,

I’m trying to change GPT2 model in the following way:  
Instead of passing sentences through the tokenizer and tokenizer output to the model, I want to use my own defined external embeddings directly as input to the model, i.e. I want to get GPT2 12 pretrained layers as a model, and pass them as input to the embeddings vector. I saw that I can extract layers by: `gpt.base_model.h`, so I tried the following code (just for example):

```auto
gpt = AutoModel.from_pretrained('gpt2')
model = nn.Sequential(*self.gpt.base_model.h)
model(torch.ones((1, 5, 768)))

```

But I get an error: `TypeError: layer_norm(): argument 'input' (position 1) must be Tensor, not tuple `, has anyone encountered this scenario and know how to fix it?

Thanks