# TPU slow finetuning T5-base
**URL:**
**Author:**  [@GenV](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGenV)%5C
**Post date:** [February 11, 2022, 3:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftpu-slow-finetuning-t5-base%2F14629%2F1 "2022-02-11T15:27:22Z")
Hi,
I am trying to train a T5-base on Colab with the TPU. I am using the official code to perform a fine-tuning on the T5-base (with my dataset), but the training with TPU is extremely slow! I’m using the offical [code](https://github.com/huggingface/transformers/blob/master/examples/pytorch/translation/run_translation.py).
I am attaching the colab code with the various libraries I have installed: [notebook](https://colab.research.google.com/drive/1g4PYEOa0zhJnMaThPZVSQepn_wr9md4f#scrollTo=XNVao-sd4fbH).
Also, if I try to increase the batch size as \>= 64, I get a memory error, as there seems to be only about 8 Gb available.
Can someone help me? Thank you!
---
**Author:**  [@finiteautomata](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffiniteautomata)%5C
**Post date:** [February 16, 2022, 1:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftpu-slow-finetuning-t5-base%2F14629%2F3 "2022-02-16T13:35:28Z")
Could you please post how you run this code? I mean, do you use `xla_spawn.py`, are you running it inline on a notebook?
---
**Author:**  [@GenV](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGenV)%5C
**Post date:** [February 16, 2022, 1:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftpu-slow-finetuning-t5-base%2F14629%2F4 "2022-02-16T13:37:51Z")
@finiteautomata I’m running it on Google colab, online with Google Colab PRO
---
**Author:**  [@finiteautomata](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffiniteautomata)%5C
**Post date:** [February 16, 2022, 1:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftpu-slow-finetuning-t5-base%2F14629%2F5 "2022-02-16T13:54:53Z")
@GenV sorry, I didn’t see your [notebook link](https://colab.research.google.com/drive/1g4PYEOa0zhJnMaThPZVSQepn_wr9md4f#scrollTo=iTXRfrAxY2RK). You are missing the `xla_spawn.py` part, that is, the code that makes your code run in a parallel fashion. You should add this:
```auto
python xla_spawn.py --num_cores 8 t5.py \
--model_name_or_path="t5-base" \
--do_train \
--do_eval \
```
etc etc
---
**Author:**  [@GenV](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGenV)%5C
**Post date:** [February 16, 2022, 2:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftpu-slow-finetuning-t5-base%2F14629%2F6 "2022-02-16T14:05:46Z")
@finiteautomata so I need [this](https://github.com/huggingface/transformers/blob/master/examples/legacy/seq2seq/xla_spawn.py) code and run
python xla\_spawn.py --num\_cores 8 t5.py \ and all other args of t5.py ?
---
**Author:**  [@finiteautomata](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffiniteautomata)%5C
**Post date:** [February 16, 2022, 2:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftpu-slow-finetuning-t5-base%2F14629%2F7 "2022-02-16T14:08:45Z")
Exactly that. Try it and tell what happens
---
**Author:**  [@GenV](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGenV)%5C
**Post date:** [February 16, 2022, 3:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftpu-slow-finetuning-t5-base%2F14629%2F8 "2022-02-16T15:36:20Z")
@finiteautomata I don’t know if it’s doing all well, I have this prints:
1- Running tokenizer on train dataset: 0% 0/30 [00:00\
**Author:**  [@GenV](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGenV)%5C
**Post date:** [February 16, 2022, 4:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftpu-slow-finetuning-t5-base%2F14629%2F9 "2022-02-16T16:25:42Z")
@finiteautomata I see that at start xla = 1, then xla = 0. Furthermore, the training seems to take a long time, about 90 hours (on gpu they become less than 2); maybe somehow it uses the CPU?
---
**Author:**  [@phosseini](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fphosseini)%5C
**Post date:** [March 2, 2022, 11:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftpu-slow-finetuning-t5-base%2F14629%2F10 "2022-03-02T23:29:28Z")
@GenV I have the same problem when using the TPU in Colab (I have Google Colab Pro +). I was not using the `xla_spawn.py`, I gave it a try and interestingly, the first time I did run my script using the `xla_spawn.py` it made my training faster, however, after terminating my node and reconnecting to the TPU, I cannot make it work again and even using the `xla_spawn.py` the training is very slow (so it was kind of random and I can’t reproduce).
Did you figure something out?
---
**Author:**  [@phosseini](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fphosseini)%5C
**Post date:** [March 3, 2022, 3:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftpu-slow-finetuning-t5-base%2F14629%2F11 "2022-03-03T03:10:37Z")
For folks who are still struggling, I think I found one potential reason for why training on TPU is slow, look [here](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhen-can-we-expect-tpu-trainer%2F10353%2F5) (I set padding to `True` in my tokenizer and I’m already seeing a speedup in my TPU training, basically it looks like it didn’t have anything to do with my torch xla installation)
---
**Author:**  [@GenV](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGenV)%5C
**Post date:** [March 3, 2022, 2:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftpu-slow-finetuning-t5-base%2F14629%2F12 "2022-03-03T14:19:52Z")
Thank you @phosseini for your answer! Yes I had the same issue with the randomness. I have read the discussion and it is interesting, so it will be for `padding = True` (I was using False if I’m not mistaken).
Also I have your same question (if you just need to put `padding = False`), I wait for answer in the other thread.
---
**Author:**  [@GenV](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGenV)%5C
**Post date:** [March 8, 2022, 9:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftpu-slow-finetuning-t5-base%2F14629%2F13 "2022-03-08T09:38:21Z")
@phosseini I tried with `pad_to_max_length=True` and it’s working fine.
---
**Author:**  [@deathcrush](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdeathcrush)%5C
**Post date:** [June 16, 2022, 8:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftpu-slow-finetuning-t5-base%2F14629%2F14 "2022-06-16T08:52:22Z")
Did you guys notice speedups vs GPU training? @GenV I have paid access to A100 GPUs but for side research tasks I’d like to use TPUs in case something works out…
---
**Author:**  [@GenV](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGenV)%5C
**Post date:** [June 17, 2022, 7:36am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftpu-slow-finetuning-t5-base%2F14629%2F15 "2022-06-17T07:36:27Z")
@deathcrush The tpu is much faster VS gpu training with a A100, P100, V100 .