# What should be used as sentence embedding for BertModel?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhat-should-be-used-as-sentence-embedding-for-bertmodel%2F6333%3E%5C
**Category:** Beginners\
**Created:** [May 24, 2021, 8:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhat-should-be-used-as-sentence-embedding-for-bertmodel%2F6333 "2021-05-24T08:56:49Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![martinmin](https://avatars.discourse-cdn.com/v4/letter/m/43a26b/32.png) [@martinmin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmartinmin)%5C
**Post date:** [May 24, 2021, 8:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhat-should-be-used-as-sentence-embedding-for-bertmodel%2F6333%2F1 "2021-05-24T08:56:49Z")

</div>

I want to get sentences’ embedding vectors for other classification tasks

```
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertModel.from_pretrained("bert-base-uncased")
inputs = tokenizer('this is a test.', return_tensors="pt")
outputs = model(**inputs)

```

If I do this way:  
`embedding_of_sentence = outputs[1]`

Here, according to the documentation, the outputs[1] is the:  
`* **pooler_output** ( `torch.FloatTensor`of shape`(batch\_size, hidden\_size)` ) – Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.`

The last layer hidden state of the first token CLS of the sentence for classification, which seems right.

However, in another post, they are suggesting using “usually only take the hidden states of the [CLS] token of the last layer”,

> <https://github.com/huggingface/transformers/issues/1950>
>
> How can I extract embeddings for a sentence or a set of words directly from pre-…trained models (Standard BERT)? For example, I am using Spacy for this purpose at the moment where I can do it as follows:
> 
> sentence vector:
> \`sentence\_vector = bert\_model("This is an apple").vector\`
> 
> word\_vectors:
> \`\`\`
> words = bert\_model("This is an apple")
> word\_vectors = \[w.vector for w in words\]
> \`\`\`
> 
> I am wondering if this is possible directly with huggingface pre-trained models (especially BERT).

and the code is:

```auto
embedding_of_last_layer = outputs[0][0]
embedding_of_sentence =embedding_of_last_layer[0]

```

The results from the two methods for the sentence embedding are different. Which one is right or better?