# Finetuning GPT-J6B for custom dataset

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetuning-gpt-j6b-for-custom-dataset%2F9924%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [September 13, 2021, 6:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetuning-gpt-j6b-for-custom-dataset%2F9924 "2021-09-13T06:54:40Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Syed313](https://avatars.discourse-cdn.com/v4/letter/s/51bf81/32.png) [@Syed313](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSyed313)%5C
**Post date:** [September 13, 2021, 6:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetuning-gpt-j6b-for-custom-dataset%2F9924%2F1 "2021-09-13T06:54:40Z")

</div>

How to prepare the dataset to feed GPT-J6B for finetuning.  
Any steps or tutorial is appreciated. Thanks

---

<div class="post-metadata">

**Author:** ![dynamicwebpaige](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dynamicwebpaige/32/4185_2.png) [@dynamicwebpaige](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdynamicwebpaige)%5C
**Post date:** [March 6, 2022, 5:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetuning-gpt-j6b-for-custom-dataset%2F9924%2F2 "2022-03-06T05:30:48Z")

</div>

Hi, @Syed313! Thanks for the question. 🙂

`@deniskamazur` modified EleutherAI’s GPT-J 6B model, so you can [generate and fine-tune it in Colab](https://github.com/huggingface/transformers/issues/14839) or on an equivalent desktop GPU (e.g. single 1080Ti).

📔 [The proof of concept notebook is available here](https://colab.research.google.com/drive/1ft6wQU0BhqG5PRlwgaZJv2VukKKjU4Es?usp=sharing#scrollTo=P8Y75B6WDIN-).

As you are probably already aware: the original GPT-J takes 22+ GB memory for `float32` parameters; and even if you cast everything to 16-bit, it will still not fit onto most single-GPU setups short of A6000 and A100. You can inference it on TPU or CPUs, but fine-tuning is a bit more expensive. This implementation should be a bit more cost-effective.