# PyTorch version
**URL:**
**Author:**  [@david-waterworth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdavid-waterworth)%5C
**Post date:** [December 15, 2020, 12:37am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpytorch-version%2F2826%2F1 "2020-12-15T00:37:19Z")
I’m having all sorts of issues training transformers on my 3090 - this card requires the cuda 11.1 which in turn requires torch 1.7.1
Is this supported? I’m using python 3.7 (I tried 3.9 but there’s no wheel for one of the dependencies for `datasets` and it wouldn’t build so I rolled back).
I installed `pip install torch==1.7.1+cu110 -f https://download.pytorch.org/whl/torch_stable.html`
Training of `RobertaForMaskedLM` frequently crashes with CUDA exceptions. If this should work I’ll create specific issues.
---
**Author:**  [@david-waterworth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdavid-waterworth)%5C
**Post date:** [December 15, 2020, 7:02am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpytorch-version%2F2826%2F2 "2020-12-15T07:02:44Z")
OK So I tracked down the crash. The problem was the position embedding. I had `max_seq_length == max_position_embeddings`, this results in a position index \> max\_position\_embeddings for any sequence which is truncated.
This is because `create_position_ids_from_input_ids` in `modeling_roberta.py' below adds pdding_idx to the cumsum - if there are no masked `input\_ids` this will be > `max\_seq\_length`
```
mask = input_ids.ne(padding_idx).int()
incremental_indices = torch.cumsum(mask, dim=1).type_as(mask) * mask
return incremental_indices.long() + padding_idx
```
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [December 15, 2020, 9:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpytorch-version%2F2826%2F3 "2020-12-15T21:25:38Z")
Hi there!
It’s hard to know exactly what’s going on without seeing your code but here is what I can share about RoBERTa. You **should not** use `max_position_embeddings` as a maximum sequence length. Because the position IDs of RoBERTa go from `padding_index` to `maximum_sequence_length + padding_index`, this `max_position_embeddings` is purposely set to 514 (2, the padding index + 512, the maximum sequence length). You should use `tokenizer.model_max_length` instead (which should be 512).
---
**Author:**  [@david-waterworth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdavid-waterworth)%5C
**Post date:** [December 16, 2020, 8:45am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpytorch-version%2F2826%2F4 "2020-12-16T08:45:48Z")
Yes that’s what I worked out, I’m not sure why the position ID’s go from `padding_index` to `maximum_sequence_length + padding_index` though (I actually think it starts at padding\_index + 1)? You could set the padding\_index of the position\_ids embedding to 0 (i.e. different to the padding index of the input\_ids) and as far as I can see it still works fine since the actual indices are always \> 0 and there’s no chance of an index out of range (which doesn’t produce a meaningful exception from CUDA)
Also did the original Bert / RoBERTa not use sinusoidal position embeddings? Or was that a later addition?
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [December 16, 2020, 3:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpytorch-version%2F2826%2F5 "2020-12-16T15:07:22Z")
This is all to mimic the original implementation of RoBERTa. So no, RoBERTa does not use sinusoidal position embeddings. That’s also why we can’t change the `padding_index` for the `posistion_ids` as it would break from the pretrained models.
---
**Author:**  [@david-waterworth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdavid-waterworth)%5C
**Post date:** [December 16, 2020, 9:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpytorch-version%2F2826%2F6 "2020-12-16T21:52:17Z")
Thanks that makes sense - although I used the value of 512 for the position embedding size based on PRETRAINED\_POSITIONAL\_EMBEDDINGS\_SIZES from [source](https://huggingface.co/transformers/_modules/transformers/models/roberta/tokenization_roberta_fast.html#RobertaTokenizerFast) - this is also the default for BertConfig from what I can see.
Plus it still crashes when max\_position\_embeddings = model\_max\_length + padding\_index, I think it should be max\_position\_embeddings = model\_max\_length + padding\_index + 1 (since the the indices start from 1 not zero)
---
**Author:**  [@david-waterworth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdavid-waterworth)%5C
**Post date:** [December 20, 2020, 11:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpytorch-version%2F2826%2F7 "2020-12-20T23:02:37Z")
One thing to note is `tokenizer.model_max_length` doesn’t work - it’s always set to int(1e30). The reason is [tokenization\_utils\_base.py#L1857](https://github.com/huggingface/transformers/blob/f38c4ad302dd34faef1137b13e9636f4408b0462/src/transformers/tokenization_utils_base.py#L1857) only sets model\_max\_length for the hard coded list of pre-trained huggingface tokenizers. If you train your own I don’t see any way of setting it from config when using AutoTokenizer().from\_pretrained()
---
**Author:**  [@TopRightExit](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTopRightExit)%5C
**Post date:** [July 12, 2022, 2:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpytorch-version%2F2826%2F8 "2022-07-12T14:31:41Z")
> [@sgugger](#):
>
> `max_position_embeddings` is purposely set to 514 (2, the padding index + 512, the maximum sequence length).
When I was using run\_mlm.py script to train a RoBERTa model, I also had the issue of not being able to set max\_seq\_length = 512. Thanks to this information on `max_position_embeddings`, I had to set --config\_overrides argument to “max\_position\_embeddings=514” when executing run\_mlm.py with max\_seq\_length = 512. The error did not show up a few training steps in. My model training is still in progress.