# Accessing uncontextualized BERT word embeddings

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccessing-uncontextualized-bert-word-embeddings%2F1812%3E%5C
**Category:** Beginners\
**Created:** [October 29, 2020, 10:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccessing-uncontextualized-bert-word-embeddings%2F1812 "2020-10-29T22:15:53Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![jdyearsley](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jdyearsley/32/535_2.png) [@jdyearsley](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjdyearsley)%5C
**Post date:** [October 29, 2020, 10:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccessing-uncontextualized-bert-word-embeddings%2F1812%2F1 "2020-10-29T22:15:53Z")

</div>

Hi there! Once I’ve imported a BERT model from HuggingFace, is there a way to convert a sequence of encoded tokens into BERT’s raw embeddings without contextualizing them using self-attention, or otherwise extract the raw embedding for a given token?

---

<div class="post-metadata">

**Author:** ![talolard](https://avatars.discourse-cdn.com/v4/letter/t/258eb7/32.png) [@talolard](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftalolard)%5C
**Post date:** [October 30, 2020, 12:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccessing-uncontextualized-bert-word-embeddings%2F1812%2F2 "2020-10-30T12:03:46Z")

</div>

Try this.  
I think the apis change a bit between models so take a look before you copy paste 🙂

```auto
            model = DistilBertForTokenClassification.from_pretrained(
                "distilbert-base-cased", num_labels=self.num_labels
            )
            word_embeddings = model.distilbert.embeddings.word_embeddings(["my token ids here"])
            word_embeddings_with_positions = model.distilbert.embeddings(["my token ids here"])

```

---

<div class="post-metadata">

**Author:** ![jdyearsley](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jdyearsley/32/535_2.png) [@jdyearsley](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjdyearsley)%5C
**Post date:** [October 30, 2020, 7:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccessing-uncontextualized-bert-word-embeddings%2F1812%2F3 "2020-10-30T19:32:14Z")

</div>

This got me there! Thank you so much.