# How padding in huggingface tokenizer works?
**URL:**
**Author:**  [@RajSingh333](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRajSingh333)%5C
**Post date:** [November 22, 2021, 5:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-padding-in-huggingface-tokenizer-works%2F12161%2F1 "2021-11-22T17:10:10Z")
I tried following tokenization example:
```
tokenizer = BertTokenizer.from_pretrained(MODEL_TYPE, do_lower_case=True)
sent = "I hate this. Not that.",
_tokenized = tokenizer(sent, padding=True, max_length=20, truncation=True)
print(_tknzr.decode(_tokenized['input_ids'][0]))
print(len(_tokenized['input_ids'][0]))
```
The output was:
```
[CLS] i hate this. not that. [SEP]
9
```
Notice the parameter to `tokenizer`: `max_length=20`. How can I make Bert tokenizer to append 11 `[PAD]` tokens to this sentence to make it total `20`?
---
**Author:**  [@adorkin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fadorkin)%5C
**Post date:** [November 22, 2021, 6:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-padding-in-huggingface-tokenizer-works%2F12161%2F2 "2021-11-22T18:38:17Z")
You need to change `padding` to `"max_length"`. The default behavior (with `padding=True`) is to pad to the length of the longest sentence in the batch, meanwhile sentences longer than specified length are getting truncated to the specified `max_length`. In your example you have only one sentence, thus there’s no padding (the only sentence is the longest one). Your sentence is shorter than max length, so there’s no truncation either.
---
**Author:**  [@RajSingh333](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRajSingh333)%5C
**Post date:** [November 22, 2021, 7:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-padding-in-huggingface-tokenizer-works%2F12161%2F3 "2021-11-22T19:28:49Z")
> [@adorkin](#):
>
> ngth, so there’s no truncation either.
Great thanks!!! It worked.
But how one can know that `padding` does indeed accept string value `max_length`? I tried to go through both of the tokenizer pages: [tokenizer](https://huggingface.co/transformers/main_classes/tokenizer.html) and [BertTokenizer](https://huggingface.co/transformers/model_doc/bert.html#berttokenizer). But none of these pages state that `padding` does indeed accept string values like `max_length`. Now I am guessing what else it might be accepting and where can I find the whole list.
Am I not reading carefully? Can you please tell how can I navigate the docs to obtain this information / link to doc stating this?
---
**Author:**  [@adorkin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fadorkin)%5C
**Post date:** [November 22, 2021, 8:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-padding-in-huggingface-tokenizer-works%2F12161%2F4 "2021-11-22T20:58:14Z")
Well, you can call `help` on an object or a specific method to see more info. For instance, `help(tokenizer. __call__ )` will display the documentation on the method that you’re using in your example. It’s the safest bet, in my opinion. However, the implementation of the method is inherited from `PreTrainedTokenizerBase` and accordingly the related docs can be found [here](https://huggingface.co/transformers/internal/tokenization_utils.html?highlight=tokenizerbase#transformers.tokenization_utils_base.PreTrainedTokenizerBase).
Although, I do agree with you that not seeing this info on the child classes’ pages may be quite confusing.
---
**Author:**  [@RajSingh333](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRajSingh333)%5C
**Post date:** [November 22, 2021, 10:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-padding-in-huggingface-tokenizer-works%2F12161%2F5 "2021-11-22T22:30:19Z")
Yes / No.
Keeping inherited stuff out of child class’ docs might reduce clutter.
But I guess it also serves some purpose like having a glance on what all things available on class without navigating whole class hierarchy. Then, a simple switch to show/hide inheritted stuff would be useful… I guess javadoc follow this approach.