# How pretrain ELECTRA on custom dataset?
**URL:**
**Author:**  [@FeryET](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FFeryET)%5C
**Post date:** [September 2, 2020, 6:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-pretrain-electra-on-custom-dataset%2F968%2F1 "2020-09-02T18:28:00Z")
I have a 1GB raw text dataset in a niche domain. I want to train an ELECTRA model but I couldn’t find any tutorial/examples to do so. Can anyone help me? I tried using the `simpletransformers` package, but it has memory issues at the moment and after a few epochs my colab session crashes.
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [September 3, 2020, 4:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-pretrain-electra-on-custom-dataset%2F968%2F2 "2020-09-03T04:15:04Z")
pinging @lysandre
---
**Author:**  [@lysandre](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flysandre)%5C
**Post date:** [September 3, 2020, 7:31am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-pretrain-electra-on-custom-dataset%2F968%2F3 "2020-09-03T07:31:47Z")
Do you want to fine-tune or pre-train an ELECTRA model? If you want to fine-tune it, you can leverage the `examples/run_language_modeling.py` script.
If you want to pre-train it, your best bet is to use the original implementation (in TF1) and then convert it to our library using our conversion script which is [here](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_electra_original_tf_checkpoint_to_pytorch.py).
[Here’s a PR](https://github.com/huggingface/transformers/pull/4656) for pre-training ELECTRA with our library but it’s not working right now and we don’t have the bandwidth to get back into it right now.
---
**Author:**  [@FeryET](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FFeryET)%5C
**Post date:** [September 3, 2020, 8:36am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-pretrain-electra-on-custom-dataset%2F968%2F4 "2020-09-03T08:36:30Z")
@lysandre I am planning to pretrain the model, but I would like to also give a shot to `examples/run_language_modeling.py` script. How can I finetune the base Electra model on MLM task on my data? The script is a bit ambigious.
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [September 6, 2020, 7:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-pretrain-electra-on-custom-dataset%2F968%2F5 "2020-09-06T07:50:22Z")
I’m not sure if fine-tuning ELECTRA with MLM is good idea since the main idea behind ELECTRA was to train it as a discriminator rather than generator and overcome the problems of MLM’s
Any thoughts @lysandre
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [September 6, 2020, 10:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-pretrain-electra-on-custom-dataset%2F968%2F6 "2020-09-06T10:53:40Z")
> [@ELECTRA training reimplementation and discussion](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Felectra-training-reimplementation-and-discussion%2F1004)%3A
>
> After months of development and debugging, I finally successfully train a model from scratch and replicate the official results. [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators](https://arxiv.org/abs/2003.10555) by Kevin Clark. Minh-Thang Luong. Quoc V. Le. Christopher D. Manning computer Code: [electra\_pytorch](https://github.com/richarddwang/electra_pytorch) AFAIK, the closest reimplementation to the original one, taking care of many easily overlooked details (described below). AFAIK, the only one successfully validate itself by replica…