# Error when Fine-tuning pretrained Masked Language Model **URL:** **Author:** ![Neel-Gupta](https://avatars.discourse-cdn.com/v4/letter/n/f1d935/32.png) [@Neel-Gupta](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNeel-Gupta)%5C **Post date:** [April 9, 2021, 11:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferror-when-fine-tuning-pretrained-masked-language-model%2F5386%2F1 "2021-04-09T11:54:09Z") My whole question is here:- [python - TypeError: zeros\_like(): argument 'input' when fine-tuning on MLM - Stack Overflow](https://stackoverflow.com/questions/67004233/typeerror-zeros-like-argument-input-when-fine-tuning-on-mlm) Basically, I am having this error when fine-tuning my pretrained model: ```auto ValueError: expected sequence of length 2033 at dim 1 (got 2036) ``` Anyone have any idea how I can solve this? ---
**Author:** ![Neel-Gupta](https://avatars.discourse-cdn.com/v4/letter/n/f1d935/32.png) [@Neel-Gupta](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNeel-Gupta)%5C **Post date:** [April 9, 2021, 11:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferror-when-fine-tuning-pretrained-masked-language-model%2F5386%2F2 "2021-04-09T23:36:13Z")
Anyone 😞 ? I have put `padding=True`, so such issues should not exist ---
**Author:** ![Maimonator](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/maimonator/32/1452_2.png) [@Maimonator](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMaimonator)%5C **Post date:** [April 11, 2021, 10:39am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferror-when-fine-tuning-pretrained-masked-language-model%2F5386%2F3 "2021-04-11T10:39:19Z")
`tokenizer(batch_sentences, padding=True)` - **padding to max sequence in batch** Maybe you wanted to use: `tokenizer(batch_sentences, padding='max_length')` - **padding to max model input length** This was taken from the docs: > **[Preprocess](https://huggingface.co/docs/transformers/main/preprocessing)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. ---
**Author:** ![Neel-Gupta](https://avatars.discourse-cdn.com/v4/letter/n/f1d935/32.png) [@Neel-Gupta](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNeel-Gupta)%5C **Post date:** [April 11, 2021, 11:59am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferror-when-fine-tuning-pretrained-masked-language-model%2F5386%2F4 "2021-04-11T11:59:35Z")
Thanks a lot for the reply @Maimonator !! I had missed putting the max\_length arg 🤗 I am getting this new error:- `ValueError: expected sequence of length 2000 at dim 1 (got 1981)` Simply put - my tokenization just function doesn’t work ☹ Can you see the code I posted on the StackOverflow link for my `tok` function and how I use the `dataset.map` to apply it to my dataset? I personally can’t figure out why it doesn’t work ---
**Author:** ![Maimonator](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/maimonator/32/1452_2.png) [@Maimonator](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMaimonator)%5C **Post date:** [April 11, 2021, 1:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferror-when-fine-tuning-pretrained-masked-language-model%2F5386%2F5 "2021-04-11T13:25:25Z")
Is this the latest tokenizer function? ```python def tok(example): encodings = tokenizer(example['src'], truncation=True, padding=True) return encodings ``` Try this instead: ```auto def tok(example): encodings = tokenizer(example['src'], truncation=True, padding="max_length", max_length=2000) return encodings ``` Let me know if this works for you ---
**Author:** ![Neel-Gupta](https://avatars.discourse-cdn.com/v4/letter/n/f1d935/32.png) [@Neel-Gupta](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNeel-Gupta)%5C **Post date:** [April 11, 2021, 2:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferror-when-fine-tuning-pretrained-masked-language-model%2F5386%2F6 "2021-04-11T14:27:18Z")
I tried with the different function, but ```auto --------------------------------------------------------------------------- IndexError Traceback (most recent call last) in () 45 ) 46 ---> 47 train_results = trainer.train() 11 frames /usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse) 1914 # remove once script supports set_grad_enabled 1915 _no_grad_embedding_renorm_(weight, input, max_norm, norm_type) -> 1916 return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) 1917 1918 IndexError: index out of range in self ``` ~~Which means there is no truncation/padding being done most probably?~~ Hmm…I removed those arguments completely to see the new error message - which explains that it indeed does truncate and pad to the model’s max input length. So apparently this error is indeed another disconnected one. ---
**Author:** ![Maimonator](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/maimonator/32/1452_2.png) [@Maimonator](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMaimonator)%5C **Post date:** [April 11, 2021, 2:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferror-when-fine-tuning-pretrained-masked-language-model%2F5386%2F7 "2021-04-11T14:51:47Z")
Maybe [this](https://huggingface.co/transformers/main_classes/model.html#transformers.PreTrainedModel.resize_token_embeddings) function is also missing? Here’s how I use it: `model.resize_token_embeddings(len(tokenizer))` ---
**Author:** ![Neel-Gupta](https://avatars.discourse-cdn.com/v4/letter/n/f1d935/32.png) [@Neel-Gupta](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNeel-Gupta)%5C **Post date:** [April 11, 2021, 2:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferror-when-fine-tuning-pretrained-masked-language-model%2F5386%2F8 "2021-04-11T14:59:59Z")
Really appreciate your replies in helping me figure out this weird problem 👍 but this gets the same error as posted above `index out of range in self`. This seems to be a pretty pesky and weird issue 😞 Wish there were more comprehensive examples on simple modelling with HF rather than the Squad and official tasks explored in these examples. ---
**Author:** ![Maimonator](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/maimonator/32/1452_2.png) [@Maimonator](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMaimonator)%5C **Post date:** [April 11, 2021, 9:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferror-when-fine-tuning-pretrained-masked-language-model%2F5386%2F9 "2021-04-11T21:41:21Z")
I hear you… I feel the examples and documentations aren’t as elaborate as we would’ve wished they’d be. Also I didn’t mention this explicitly, but I’ve set `max_length=2000` in this tokenization function: ```auto def tok(example): encodings = tokenizer(example['src'], truncation=True, padding="max_length", max_length=2000) return encodings ``` But you should set it to whatever you think is legit. I don’t have any new ideas as I’m quite new to this library as well, but update us on the development! Hopefully you’ll solve it soon 😊 ---
**Author:** ![Neel-Gupta](https://avatars.discourse-cdn.com/v4/letter/n/f1d935/32.png) [@Neel-Gupta](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNeel-Gupta)%5C **Post date:** [April 13, 2021, 7:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferror-when-fine-tuning-pretrained-masked-language-model%2F5386%2F10 "2021-04-13T19:56:41Z")
day 100 of reporting, still getting this error ☹ ```auto --------------------------------------------------------------------------- IndexError Traceback (most recent call last) in () 47 ) 48 ---> 49 train_results = trainer.train() 11 frames /usr/local/lib/python3.7/dist-packages/transformers/trainer.py in train(self, resume_from_checkpoint, trial, **kwargs) 1118 tr_loss += self.training_step(model, inputs) 1119 else: -> 1120 tr_loss += self.training_step(model, inputs) 1121 self._total_flos += float(self.floating_point_ops(inputs)) 1122 /usr/local/lib/python3.7/dist-packages/transformers/trainer.py in training_step(self, model, inputs) 1522 loss = self.compute_loss(model, inputs) 1523 else: -> 1524 loss = self.compute_loss(model, inputs) 1525 1526 if self.args.n_gpu > 1: /usr/local/lib/python3.7/dist-packages/transformers/trainer.py in compute_loss(self, model, inputs, return_outputs) 1554 else: 1555 labels = None -> 1556 outputs = model(**inputs) 1557 # Save past state if it exists 1558 # TODO: this needs to be fixed and made cleaner later. /usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs) 887 result = self._slow_forward(*input, **kwargs) 888 else: --> 889 result = self.forward(*input, **kwargs) 890 for hook in itertools.chain( 891 _global_forward_hooks.values(), /usr/local/lib/python3.7/dist-packages/transformers/models/longformer/modeling_longformer.py in forward(self, input_ids, attention_mask, global_attention_mask, head_mask, token_type_ids, position_ids, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict) 1855 output_attentions=output_attentions, 1856 output_hidden_states=output_hidden_states, -> 1857 return_dict=return_dict, 1858 ) 1859 sequence_output = outputs[0] /usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs) 887 result = self._slow_forward(*input, **kwargs) 888 else: --> 889 result = self.forward(*input, **kwargs) 890 for hook in itertools.chain( 891 _global_forward_hooks.values(), /usr/local/lib/python3.7/dist-packages/transformers/models/longformer/modeling_longformer.py in forward(self, input_ids, attention_mask, global_attention_mask, head_mask, token_type_ids, position_ids, inputs_embeds, output_attentions, output_hidden_states, return_dict) 1662 1663 embedding_output = self.embeddings( -> 1664 input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds 1665 ) 1666 /usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs) 887 result = self._slow_forward(*input, **kwargs) 888 else: --> 889 result = self.forward(*input, **kwargs) 890 for hook in itertools.chain( 891 _global_forward_hooks.values(), /usr/local/lib/python3.7/dist-packages/transformers/models/longformer/modeling_longformer.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds) 491 if inputs_embeds is None: 492 inputs_embeds = self.word_embeddings(input_ids) --> 493 position_embeddings = self.position_embeddings(position_ids) 494 token_type_embeddings = self.token_type_embeddings(token_type_ids) 495 /usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs) 887 result = self._slow_forward(*input, **kwargs) 888 else: --> 889 result = self.forward(*input, **kwargs) 890 for hook in itertools.chain( 891 _global_forward_hooks.values(), /usr/local/lib/python3.7/dist-packages/torch/nn/modules/sparse.py in forward(self, input) 156 return F.embedding( 157 input, self.weight, self.padding_idx, self.max_norm, --> 158 self.norm_type, self.scale_grad_by_freq, self.sparse) 159 160 def extra_repr(self) -> str: /usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse) 1914 # remove once script supports set_grad_enabled 1915 _no_grad_embedding_renorm_(weight, input, max_norm, norm_type) -> 1916 return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) 1917 1918 IndexError: index out of range in self ``` Still trying to get out of this… ---
**Author:** ![PriyankaDilip](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/priyankadilip/32/2996_2.png) [@PriyankaDilip](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPriyankaDilip)%5C **Post date:** [October 29, 2021, 12:47am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferror-when-fine-tuning-pretrained-masked-language-model%2F5386%2F11 "2021-10-29T00:47:59Z")
This worked for me using the length in the error message I was getting, thanks Maimonator! ---
**Author:** ![AlexKay](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alexkay/32/33629_2.png) [@AlexKay](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAlexKay)%5C **Post date:** [November 24, 2021, 3:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferror-when-fine-tuning-pretrained-masked-language-model%2F5386%2F12 "2021-11-24T15:37:17Z")
seems like this one fixed my problem! thanks ---
**Author:** ![pchhapolika](https://avatars.discourse-cdn.com/v4/letter/p/b9e5f3/32.png) [@pchhapolika](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpchhapolika)%5C **Post date:** [March 9, 2023, 8:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferror-when-fine-tuning-pretrained-masked-language-model%2F5386%2F13 "2023-03-09T08:40:09Z")
Hi @AlexKay How did you fix this issue?