# How to use TPU for BERT training Colab

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-tpu-for-bert-training-colab%2F16228%3E%5C
**Category:** Beginners\
**Created:** [March 29, 2022, 5:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-tpu-for-bert-training-colab%2F16228 "2022-03-29T17:05:20Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![silkski](https://avatars.discourse-cdn.com/v4/letter/s/e9bcb4/32.png) [@silkski](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsilkski)%5C
**Post date:** [March 29, 2022, 5:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-tpu-for-bert-training-colab%2F16228%2F1 "2022-03-29T17:05:20Z")

</div>

Hi there,

I’m trying to further pre-train BERT model on my own very large dataset using the `run_mlm.py` file in `transformers/examples/pytorch/language-modelling/`. (sorry for not providing link - 2 link limit)

I’m using a Colab TPU to speed training, however training times seem to be incredibly long.

Here’s the code I’m using for training:

```auto
python /content/transformers/examples/legacy/seq2seq/xla_spawn.py --num_cores 8 /content/transformers/examples/pytorch/language-modeling/run_mlm.py \
--model_name_or_path bert-base-cased \
--validation_split_percentage 20 \
--line_by_line \
--do_train \
--do_eval \
--tpu_num_cores 8 \
--learning_rate 2e-5 \
--per_device_train_batch_size 64 \
--per_device_eval_batch_size 128 \
--num_train_epochs 4 \
--output_dir /content/output \
--train_file /content/text.txt \
--overwrite_output_dir

```

I installed torch, torch\_xla and set `XRT_TPU_CONFIG `according to [this](https://stackoverflow.com/a/69143183).

I installed transfomers from source as required.

transformers version `4.18.0.dev0`  
pytorch version `1.8.2+cpu`

That’s full [output](https://pastebin.com/cKJymAnk) I’m getting.

As you can see estimated training time is 354 hours, which is ridiculously long. I’m guessing I’m not loading the TPU correctly and as a result training is running on CPU, however I’m not sure what’s the correct way to do that.

Any help is much appreciated. Thanks.

---

<div class="post-metadata">

**Author:** ![Enes3774](https://avatars.discourse-cdn.com/v4/letter/e/c5a1d2/32.png) [@Enes3774](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FEnes3774)%5C
**Post date:** [July 30, 2022, 2:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-tpu-for-bert-training-colab%2F16228%2F2 "2022-07-30T14:03:20Z")

</div>

hi,did you solve problem. I have the same problem. Please answer