# How to continue training on another dataset?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-continue-training-on-another-dataset%2F16780%3E%5C
**Category:** 🤗Transformers\
**Created:** [April 13, 2022, 7:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-continue-training-on-another-dataset%2F16780 "2022-04-13T07:33:21Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![beyond](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/beyond/32/28447_2.png) [@beyond](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbeyond)%5C
**Post date:** [April 13, 2022, 7:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-continue-training-on-another-dataset%2F16780%2F1 "2022-04-13T07:33:21Z")

</div>

Hi, I want to do some language model pre-training, using the Trainer API.

Assume I have two datasets `wikitext` and `bookcorpus`. I want to first train on `wikitext` and then on `bookcorpus`, and I want to save the checkpoint after training on `wikitext`, then continue training on `bookcorpus` and save the later checkpoints.

I wish to have the checkpoints something like this:

```auto
checkpoint-500 (only wikitext)
checkpoint-1000 (only wikitext)
checkpoint-1500 (only wikitext)
checkpoint-1800 (finished training on wikitext)
checkpoint-2300 (continue training on bookcorpus)
...

```

I don’t want to mix the two datasets together, because I want to analyse what’s the difference after training on another dataset. I want to know how to achieve this?

Could anyone help me?

---

<div class="post-metadata">

**Author:** ![beyond](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/beyond/32/28447_2.png) [@beyond](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbeyond)%5C
**Post date:** [April 14, 2022, 3:12pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-continue-training-on-another-dataset%2F16780%2F2 "2022-04-14T15:12:55Z")

</div>

@sgugger Could you please have a look?