# Further pre-train language model in transformers like BERT

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-pre-train-language-model-in-transformers-like-bert%2F2198%3E%5C
**Category:** Models\
**Created:** [November 25, 2020, 7:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-pre-train-language-model-in-transformers-like-bert%2F2198 "2020-11-25T07:55:11Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![lizzzi111](https://avatars.discourse-cdn.com/v4/letter/l/41988e/32.png) [@lizzzi111](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flizzzi111)%5C
**Post date:** [November 25, 2020, 7:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-pre-train-language-model-in-transformers-like-bert%2F2198%2F1 "2020-11-25T07:55:11Z")

</div>

Hi all,  
yesterday through a workshop I learned about this forum. So I have the following question: is it possible to further pre-train transformers (e.g. BERT, DistilBert) using my own corpus? I mean not for the downstream task, but the language model (e.g. BERT tasks MSM and NSP) itself? Is it possible in general and with huggingface specifically?

Thank you. best regards  
LIza

---

<div class="post-metadata">

**Author:** ![thomwolf](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thomwolf/32/2198_2.png) [@thomwolf](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fthomwolf)%5C
**Post date:** [November 25, 2020, 11:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-pre-train-language-model-in-transformers-like-bert%2F2198%2F2 "2020-11-25T11:29:57Z")

</div>

Hi @lizzzi111, nice to see you here 🙂

Yes it’s possible.

Examples and readme to do so are here: [https://github.com/huggingface/transformers/tree/master/examples/language-modeling](https://github.com/huggingface/transformers/tree/master/examples/language-modeling)

---

<div class="post-metadata">

**Author:** ![lizzzi111](https://avatars.discourse-cdn.com/v4/letter/l/41988e/32.png) [@lizzzi111](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flizzzi111)%5C
**Post date:** [November 25, 2020, 11:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-pre-train-language-model-in-transformers-like-bert%2F2198%2F3 "2020-11-25T11:46:34Z")

</div>

Thank you a lot, @thomwolf!

---

<div class="post-metadata">

**Author:** ![musitafa](https://avatars.discourse-cdn.com/v4/letter/m/e68b1a/32.png) [@musitafa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmusitafa)%5C
**Post date:** [March 27, 2022, 7:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-pre-train-language-model-in-transformers-like-bert%2F2198%2F4 "2022-03-27T19:49:18Z")

</div>

Hi @thomwolf ,

The link is expired, could you please send the link again?

Thank you