# Tokenizing two sentences with the tokenizer

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizing-two-sentences-with-the-tokenizer%2F10858%3E%5C
**Category:** Beginners\
**Created:** [October 18, 2021, 1:49am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizing-two-sentences-with-the-tokenizer%2F10858 "2021-10-18T01:49:36Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Seohyeong](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/seohyeong/32/2273_2.png) [@Seohyeong](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSeohyeong)%5C
**Post date:** [October 18, 2021, 1:49am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizing-two-sentences-with-the-tokenizer%2F10858%2F1 "2021-10-18T01:49:37Z")

</div>

Hello, everyone.

I’m working on an NLI task (such as MNLI, RTE, etc.) where two sentences are given to predict if the first sentence entails the second one or not. I’d like to know how the huggingface tokenizer behaves when the length of the first sentence exceeds the maximum sequence length of the model.

I’m using `encode_plus()` to tokenize my sentences as follows:  
`inputs = tokenizer.encode_plus(example.text_a, example.text_b, add_special_tokens=True, max_length=max_length,)`

I’d like to avoid the case of the second sentence not being encoded since the first sentence itself already exceeds the maximum input sequence length of the model. Is there an option for the `encode_plus()` function to truncate the first sentence to make sure I always have the second one in the processed data?

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [October 18, 2021, 7:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizing-two-sentences-with-the-tokenizer%2F10858%2F2 "2021-10-18T07:33:28Z")

</div>

Hi,

As explained in the [docs](https://huggingface.co/transformers/preprocessing.html#everything-you-always-wanted-to-know-about-padding-and-truncation), you can specify several possible strategies for the truncation parameter, including `'only_first'`. Also, the `encode_plus` method is outdated actually. It is recommended to just call the tokenizer, both on single sentence or pair of sentences. TLDR:

`inputs = tokenizer(text_a, text_b, truncation='only_first', max_length=max_length)`