# How to use T5 for sentence embedding?
**URL:**
**Author:**  [@banucool](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbanucool)%5C
**Post date:** [September 12, 2020, 12:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-t5-for-sentence-embedding%2F1097%2F1 "2020-09-12T12:11:23Z")
is there any way to use encoder part of T5 model for representation learning?
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [September 12, 2020, 1:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-t5-for-sentence-embedding%2F1097%2F2 "2020-09-12T13:11:28Z")
Hi @banucool
You can initialize the `T5Model` class and only forward pass through itβs encoder. The first element of the returned tuple is the final hidden states.
```python
model = T5Model.from_pretrained("t5-small")
tok = T5Tokenizer.from_pretrained("t5-small")
enc = tok("some text", return_tensors="pt")
# forward pass through encoder only
output = model.encoder(
input_ids=enc["input_ids"],
attention_mask=enc["attention_mask"],
return_dict=True
)
# get the final hidden states
emb = output.last_hidden_state
```
The shape of `emb` will be `(batch_size, seq_len, hidden_size)`
---
**Author:**  [@banucool](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbanucool)%5C
**Post date:** [September 12, 2020, 2:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-t5-for-sentence-embedding%2F1097%2F3 "2020-09-12T14:06:41Z")
thanks a lot @valhalla π
---
**Author:**  [@banucool](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbanucool)%5C
**Post date:** [September 12, 2020, 3:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-t5-for-sentence-embedding%2F1097%2F4 "2020-09-12T15:33:12Z")
can we use pruned version of bert for feature extraction?does it make sense?
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [September 12, 2020, 6:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-t5-for-sentence-embedding%2F1097%2F5 "2020-09-12T18:08:55Z")
To clarify, the above code just returns the final hidden state of each token and not whole sentence embedding.
for sentence embedding you can try [sentence-bert](https://github.com/UKPLab/sentence-transformers).
[https://huggingface.co/sentence-transformers](https://huggingface.co/sentence-transformers)
---
**Author:**  [@mriggs](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmriggs)%5C
**Post date:** [May 31, 2022, 5:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-t5-for-sentence-embedding%2F1097%2F6 "2022-05-31T17:01:27Z")
> [@valhalla](#):
>
> ```auto
> model = T5Model.from_pretrained("t5-small")
> tok = T5Tokenizer.from_pretrained("t5-small")
>
> enc = tok("some text", return_tensors="pt")
>
> # forward pass through encoder only
> output = model.encoder(
> input_ids=enc["input_ids"],
> attention_mask=enc["attention_mask"],
> return_dict=True
> )
> # get the final hidden states
> emb = output.last_hidden_state
>
> ```
Hi, Iβm interested in using T5 to generate word embeddings. I tried the code supplied above. Unfortunately, got this error message:
```auto
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
in ()
1 model = T5Model.from_pretrained("t5-small")
----> 2 tok = T5Tokenizer.from_pretrained("t5-small")
3
4 enc = tok("some text", return_tensors="pt")
5
TypeError: 'NoneType' object is not callable
```
Do you have any thoughts on resolving this error message?
Thank you in advance for your help. π
---
**Author:**  [@guotong1988](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fguotong1988)%5C
**Post date:** [May 27, 2023, 2:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-t5-for-sentence-embedding%2F1097%2F7 "2023-05-27T02:25:02Z")
> **[Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://arxiv.org/abs/2108.08877)**
>
> We provide the first exploration of sentence embeddings from text-to-text
> transformers (T5). Sentence embeddings are broadly useful for language
> processing tasks. While T5 achieves impressive performance on language tasks
> cast as sequence-to-sequence...