# Transformers on GCP Training stuck on start

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-on-gcp-training-stuck-on-start%2F33032%3E%5C
**Category:** 🤗Transformers\
**Created:** [March 4, 2023, 4:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-on-gcp-training-stuck-on-start%2F33032 "2023-03-04T16:36:02Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![PhilW](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philw/32/17254_2.png) [@PhilW](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPhilW)%5C
**Post date:** [March 4, 2023, 4:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-on-gcp-training-stuck-on-start%2F33032%2F1 "2023-03-04T16:36:02Z")

</div>

Hi everyone,

I have a strange problem with running transformer models on Google Cloud VM instances.

Once training should start, the program is stuck without an error message at  
\*\*\*\*\* Running training \*\*\*\*\*  
Num examples = 6838  
Num Epochs = 5  
Instantaneous batch size per device = 8  
Total train batch size (w. parallel, distributed & accumulation) = 8  
Gradient Accumulation steps = 1  
Total optimization steps = 4275  
Number of trainable parameters = 109490699  
0%| | 0/4275 [00:00\<?, ?it/s]

nvidia-smi shows that the GPU is idle, top shows the process idle.

I thought I might have an error with my scripts, but even when I try one of the examples ( i.e., [Google Colab](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/BERT/Fine_tuning_BERT_(and_friends)_for_multi_label_text_classification.ipynb) ) the same error occurs.

In Colab, the scripts run without a problem. The error occurs with “deep learning VM” image and barebone Linux VM. System, python packages, all up-to-date.

I have seen some postings around the web with similar problems but no real solution. I wonder if someone figured it out?

---

<div class="post-metadata">

**Author:** ![agademic](https://avatars.discourse-cdn.com/v4/letter/a/e9a140/32.png) [@agademic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fagademic)%5C
**Post date:** [March 17, 2023, 4:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-on-gcp-training-stuck-on-start%2F33032%2F2 "2023-03-17T16:09:24Z")

</div>

I am running into a similar issue (also on GCP). Even though I am using a different [script](https://github.com/lvwerra/trl/blob/main/examples/sentiment/scripts/gpt-neox-20b_peft/clm_finetune_peft_imdb.py). Training is stuck immediately.

Did you find any solution?

---

<div class="post-metadata">

**Author:** ![agademic](https://avatars.discourse-cdn.com/v4/letter/a/e9a140/32.png) [@agademic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fagademic)%5C
**Post date:** [March 17, 2023, 4:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-on-gcp-training-stuck-on-start%2F33032%2F3 "2023-03-17T16:35:16Z")

</div>

This was the [solution](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-train-stalls-when-using-tutorial-example-on-gcp-instance%2F29132%2F2) for me.

---

<div class="post-metadata">

**Author:** ![PhilW](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philw/32/17254_2.png) [@PhilW](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPhilW)%5C
**Post date:** [March 22, 2023, 6:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformers-on-gcp-training-stuck-on-start%2F33032%2F4 "2023-03-22T18:41:41Z")

</div>

As agademic wrote, uninstalling torch\_xla did the trick for me.