# How pretrain ELECTRA on custom dataset?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-pretrain-electra-on-custom-dataset%2F968%3E%5C
**Category:** Beginners\
**Created:** [September 2, 2020, 6:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-pretrain-electra-on-custom-dataset%2F968 "2020-09-02T18:27:59Z")\
**Posts on this page:** 1\
**Showing post:** 6

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [September 6, 2020, 10:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-pretrain-electra-on-custom-dataset%2F968%2F6 "2020-09-06T10:53:40Z")

</div>

> [@ELECTRA training reimplementation and discussion](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Felectra-training-reimplementation-and-discussion%2F1004)%3A
>
> After months of development and debugging, I finally successfully train a model from scratch and replicate the official results. [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators](https://arxiv.org/abs/2003.10555) by Kevin Clark. Minh-Thang Luong. Quoc V. Le. Christopher D. Manning computer Code: [electra\_pytorch](https://github.com/richarddwang/electra_pytorch) AFAIK, the closest reimplementation to the original one, taking care of many easily overlooked details (described below). AFAIK, the only one successfully validate itself by replica…

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-pretrain-electra-on-custom-dataset%2F968)._