# Pool \[CLS\] token from DistilBERT

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpool-cls-token-from-distilbert%2F11238%3E%5C
**Category:** 🤗Transformers\
**Created:** [October 31, 2021, 11:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpool-cls-token-from-distilbert%2F11238 "2021-10-31T23:29:24Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![AndreGodinho](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/andregodinho/32/3005_2.png) [@AndreGodinho](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAndreGodinho)%5C
**Post date:** [October 31, 2021, 11:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpool-cls-token-from-distilbert%2F11238%2F1 "2021-10-31T23:29:24Z")

</div>

Hey there everyone, I haven’t used this amazing API for a while. The code below is able to pool the [CLS] output embedding from DistilBERT?

```auto
distilbert = DistilBertModel.from_pretrained('distilbert-base-uncased', return_dict=True)

# more code ...

# input_ids & attention_mask from a batch
outputs = distilbert(input_ids, attention_mask=attention_mask)
seq_embeddings = outputs[0] # DistilBERT output embeddings 
cls_embedding = seq_embeddings[:,0,:] # <--- [CLS] output embedding

```

I’ve taken this idea from the following DPR code:

> <https://github.com/huggingface/transformers/blob/master/src/transformers/models/dpr/modeling_dpr.py#L204>

---

<div class="post-metadata">

**Author:** ![toyl](https://avatars.discourse-cdn.com/v4/letter/t/73ab20/32.png) [@toyl](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftoyl)%5C
**Post date:** [January 18, 2022, 8:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpool-cls-token-from-distilbert%2F11238%2F2 "2022-01-18T20:34:32Z")

</div>

did you get a solution for your question, please