# Pretrain GPT2 from scratch in Korean

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-korean%2F7832%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [July 1, 2021, 12:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-korean%2F7832 "2021-07-01T12:37:05Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![TekFell](https://avatars.discourse-cdn.com/v4/letter/t/a9a28c/32.png) [@TekFell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTekFell)%5C
**Post date:** [July 1, 2021, 12:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-korean%2F7832%2F1 "2021-07-01T12:37:05Z")

</div>

# Korean-GPT

There are already published Korean GPT2 models, but these models are limited in their use in the industry due to license restrictions. Recently, a large number of South Korean public natural language datasets such as KLUE dataset, AI-Hub dataset, and Modu Corpus have been additionally released. My goal is to train a Korean GPT2 model that anyone can use for any purpose, using only large publicly accessible datasets, including recently added datasets.

## 2. Language

The model will be trained in Korean

## 3. Model

GPT2

## 4. Datasets

KLUE Dataset  
KorQuAD Dataset  
Modu Corpus(모두의 말뭉치)  
Korean Text Datasets from AI-hub  
+ Maybe extra publicly accessible Korean datasets…

- [https://klue-benchmark.com](https://klue-benchmark.com)
- [https://korquad.github.io](https://korquad.github.io)
- [https://corpus.korean.go.kr](https://corpus.korean.go.kr)
- [https://aihub.or.kr](https://aihub.or.kr)

## 5. Training scripts

A causal language modeling script for Flax is available [here](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_clm_flax.py)

## 6. Challenges

It will take time and effort to preprocess each of the various public Korean datasets in the most helpful form for pre-training.

---

<div class="post-metadata">

**Author:** ![ben9004](https://avatars.discourse-cdn.com/v4/letter/b/848f3c/32.png) [@ben9004](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fben9004)%5C
**Post date:** [July 1, 2021, 12:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-korean%2F7832%2F2 "2021-07-01T12:54:39Z")

</div>

I think this will be an amazing challenge! Wish I can join this project!

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [July 1, 2021, 11:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-korean%2F7832%2F3 "2021-07-01T23:55:22Z")

</div>

Awesome finalizing this project 🙂

---

<div class="post-metadata">

**Author:** ![TekFell](https://avatars.discourse-cdn.com/v4/letter/t/a9a28c/32.png) [@TekFell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTekFell)%5C
**Post date:** [July 16, 2021, 4:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-korean%2F7832%2F4 "2021-07-16T16:29:54Z")

</div>

! important

Among the above datasets, I will correct the dataset that I incorrectly labeled as Modu to the official name "NIKL Corpus(국립국어원 모두의 말뭉치) ".