# All the training jobs end up getting stopped

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fall-the-training-jobs-end-up-getting-stopped%2F21031%3E%5C
**Category:** 🤗AutoTrain\
**Created:** [July 30, 2022, 8:47am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fall-the-training-jobs-end-up-getting-stopped%2F21031 "2022-07-30T08:47:03Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Blaise-g](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/blaise-g/32/5536_2.png) [@Blaise-g](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBlaise-g)%5C
**Post date:** [July 30, 2022, 8:47am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fall-the-training-jobs-end-up-getting-stopped%2F21031%2F1 "2022-07-30T08:47:03Z")

</div>

I was trying out the autotrain platform by fine tuning a model on a dataset (that was related but different to the one which was used for the fine tuning of the previous checkpoints) and I don’t understand why all models were stopped after hours of training regardless of the performance achieved in terms of metrics (with some models showing significant better results than others).

Do you have any idea on what might be going on? I have also tried to reach you guys at [autonlp@huggingface.co](mailto:autonlp@huggingface.co) with some more details about the id of the project but figured it was worth asking here as well in case anyone else was experiencing similar issues.

Thank you in advance for your time!  
getting stopped after

---

<div class="post-metadata">

**Author:** ![ben-yu](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ben-yu/32/5548_2.png) [@ben-yu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fben-yu)%5C
**Post date:** [July 31, 2022, 7:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fall-the-training-jobs-end-up-getting-stopped%2F21031%2F2 "2022-07-31T19:29:51Z")

</div>

I’m experiencing the same thing. It would be great if the interface surfaced at least an error reason

---

<div class="post-metadata">

**Author:** ![maximus12793](https://avatars.discourse-cdn.com/v4/letter/m/e9bcb4/32.png) [@maximus12793](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmaximus12793)%5C
**Post date:** [September 6, 2022, 5:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fall-the-training-jobs-end-up-getting-stopped%2F21031%2F3 "2022-09-06T17:43:43Z")

</div>

+1 Seems that after 62000 steps my training stops for no reason.

---

<div class="post-metadata">

**Author:** ![kaane0202](https://avatars.discourse-cdn.com/v4/letter/k/90db22/32.png) [@kaane0202](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkaane0202)%5C
**Post date:** [April 20, 2023, 12:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fall-the-training-jobs-end-up-getting-stopped%2F21031%2F4 "2023-04-20T12:48:41Z")

</div>

Have you found a solution? My training also stops without any errors or logs

---

<div class="post-metadata">

**Author:** ![rfisherphotos](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@rfisherphotos](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frfisherphotos)%5C
**Post date:** [July 15, 2023, 4:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fall-the-training-jobs-end-up-getting-stopped%2F21031%2F5 "2023-07-15T16:26:24Z")

</div>

Same problem here. No errors or stack traces – any way to access the logs?

---

<div class="post-metadata">

**Author:** ![michellehbn](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/michellehbn/32/58964_2.png) [@michellehbn](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmichellehbn)%5C
**Post date:** [September 27, 2023, 11:20am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fall-the-training-jobs-end-up-getting-stopped%2F21031%2F6 "2023-09-27T11:20:06Z")

</div>

Hi! Thanks for reporting and sorry for the wait. It can be expected for a Space to restart from time to time if not enough RAM and memory for the training. In cases like this, we recommend using a larger instance to help. Please let us know though if there’s any other questions! Thanks again

---

<div class="post-metadata">

**Author:** ![Adilmar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/adilmar/32/52970_2.png) [@Adilmar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAdilmar)%5C
**Post date:** [April 17, 2024, 2:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fall-the-training-jobs-end-up-getting-stopped%2F21031%2F7 "2024-04-17T14:10:35Z")

</div>

@michellehbn Can this happen occasionally, and would the solution be to restart the self-training?
