# Identical CLS token embeddings for all different sentences?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fidentical-cls-token-embeddings-for-all-different-sentences%2F8258%3E%5C
**Category:** Beginners\
**Created:** [July 11, 2021, 11:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fidentical-cls-token-embeddings-for-all-different-sentences%2F8258 "2021-07-11T23:14:30Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![qpazuzu](https://avatars.discourse-cdn.com/v4/letter/q/0ea827/32.png) [@qpazuzu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fqpazuzu)%5C
**Post date:** [July 11, 2021, 11:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fidentical-cls-token-embeddings-for-all-different-sentences%2F8258%2F1 "2021-07-11T23:14:30Z")

</div>

Hi, I was trying to extract CLS token embeddings for my encoded inputs over a loop. I see the embeddings of CLS tokens for all my sentences are identical. Can someone please tell me what I am missing here or is this expected?

Code snippet

```auto
model_chkpt = 'roberta-large'
model = AutoModelForSequenceClassification.from_pretrained(model_chkpt, output_hidden_states=True).to('cuda')

embeddings = []
with torch.no_grad():
    for _, record in enumerate(ds_enc.shard(index=1, num_shards=1000)):
        inputs = {k: torch.unsqueeze(v, dim=1).to('cuda') for k,v in record.items()}
        outputs = model(**inputs)
        #extract CLS token from the embeddings layer
        cls_embedding = torch.squeeze(outputs.hidden_states[0][0])
        embeddings.append(cls_embedding.detach().cpu().numpy())

#Prints true for all the iteration
for _, i in enumerate(embeddings):
    print((embeddings[0] == i).all())

```

From what I have read CLS token for each text input is different. I am not sure if my understanding is correct. If my understanding is correct then I am seriously doing something wrong that I am not able to figure out.

Can someone please let me know where I am going wrong.

---

<div class="post-metadata">

**Author:** ![nomorewords](https://avatars.discourse-cdn.com/v4/letter/n/d78d45/32.png) [@nomorewords](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnomorewords)%5C
**Post date:** [April 17, 2023, 4:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fidentical-cls-token-embeddings-for-all-different-sentences%2F8258%2F2 "2023-04-17T16:00:34Z")

</div>

I’m having the same issue. Did you figure it out? Thanks!