# Training DistilGPT2
**URL:**
**Author:**  [@abhilashpal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabhilashpal)%5C
**Post date:** [July 18, 2020, 3:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-distilgpt2%2F372%2F1 "2020-07-18T15:28:12Z")
Hello!
I am trying to find resources/code samples to retrain the DistilGPT2 model with text I have preprocessed myself, but could not find any. Most of the documentation relates to DistilBert and it’s uses.
Furthermore, I also have trained a gpt2-simple (tensorflow based) model. If there is any way to distil the same, it will help me too!
Thanks for your help.
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [July 18, 2020, 4:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-distilgpt2%2F372%2F2 "2020-07-18T16:43:42Z")
Hi @abhilashpal, you can find distillation code [here](https://github.com/huggingface/transformers/tree/master/examples/distillation). The same script that produces distillbert can be used for GPT-2, it’s not documented though.
you should be able to use this command after processing your dataset
```bash
python train.py \
--student_type gpt2 \
--student_config training_configs/distilgpt2.json \
--teacher_type gpt2 \
--teacher_name gpt2 \ # or your own teacher model
--alpha_ce 5.0 --alpha_cos 1.0 --alpha_clm 0.5 \
--freeze_pos_embs \
--dump_path serialization_dir/my_first_training \
--data_file data/binarized_text.bert-base-uncased.pickle \ # your data path
--token_counts data/token_counts.bert-base-uncased.pickle \ # your own pickle file path
--force # overwrites the `dump_path` if it already exists.
```
pinging @julien-c for more info
---
**Author:**  [@abhilashpal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabhilashpal)%5C
**Post date:** [July 19, 2020, 2:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-distilgpt2%2F372%2F3 "2020-07-19T14:07:58Z")
Thanks for the reply. I ran into one problem whilst running the distilbert binarization. Can anyone tell me if this means that my data per line exceeds what the distilbert model expects?
```
!python scripts/binarized_data.py \
--file_path data/dataemail.txt \
--tokenizer_type bert \
--tokenizer_name bert-base-uncased \
--dump_file data/binarized_text
```
07/19/2020 07:25:56 - WARNING - transformers.tokenization\_utils\_base - Token indices sequence length is longer than the specified maximum sequence length for this model (626 \> 512). Running this sequence through the model will result in indexing errors
---
**Author:**  [@aclifton314](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Faclifton314)%5C
**Post date:** [July 20, 2020, 8:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-distilgpt2%2F372%2F4 "2020-07-20T20:59:56Z")
@abhilashpal, that’s what it looks like to me. I don’t have any direct experience with the distillation examples, but I took a quick look at the Distillbert and Bert paper and 512 looks like their max token length.
---
**Author:**  [@eddieone](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Feddieone)%5C
**Post date:** [October 13, 2020, 9:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-distilgpt2%2F372%2F5 "2020-10-13T21:50:58Z")
I ran the previous command mentioned on the readme and it did output the binary file. However, another file is missing, `token_counts.distilgpt2.pickle `. How to generate that file as well?
On the token counter script, it returns `IndexError: list assignment index out of range` Someone know how to resolve that error?