# Train gpt-2 from scratch in Italian

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-gpt-2-from-scratch-in-italian%2F22772%3E%5C
**Category:** Beginners\
**Created:** [September 8, 2022, 2:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-gpt-2-from-scratch-in-italian%2F22772 "2022-09-08T14:29:05Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![giannigi](https://avatars.discourse-cdn.com/v4/letter/g/bbe5ce/32.png) [@giannigi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgiannigi)%5C
**Post date:** [September 8, 2022, 2:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-gpt-2-from-scratch-in-italian%2F22772%2F1 "2022-09-08T14:29:05Z")

</div>

Hello, I’m looking for a guide or some help to train gpt-2 from scratch on a small corpus. If this makes any difference the corpus is in italian, and I only know pytorch.

About tokenizers: is the tokenizer somehow language-agnostic?

About the corpus: is it supposed to be formatted in a specific way? I’ve seen some corpus’ files formatted with a space surrounding all punctuation symbols (before and after).

Is this the recomended script?

> <https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm.py>

or this??

> **[Google Colaboratory](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/causal_language_modeling_flax.ipynb)**

I’ve found this guide but uses tensorflow???

> **[Train GPT-2 in your own language](https://towardsdatascience.com/train-gpt-2-in-your-own-language-fc6ad4d60171)**
>
> A step-by-step guide to train your own GPT-2 model for text generation in your choice of language from scratch

> **[Training a GPT-2 Model From Scratch - aitextgen](https://docs.aitextgen.io/tutorials/model-from-scratch/)**
>
> A robust Python tool for text-based AI training and generation using GPT-2 and GPT Neo.

thanks