# How to fine tune fine tune GitHub Copilot?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-fine-tune-fine-tune-github-copilot%2F18889%3E%5C
**Category:** Research\
**Created:** [June 9, 2022, 4:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-fine-tune-fine-tune-github-copilot%2F18889 "2022-06-09T04:26:50Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![neo-benjamin](https://avatars.discourse-cdn.com/v4/letter/n/f6c823/32.png) [@neo-benjamin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fneo-benjamin)%5C
**Post date:** [June 9, 2022, 4:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-fine-tune-fine-tune-github-copilot%2F18889%2F1 "2022-06-09T04:26:50Z")

</div>

We can fine tune language models like BERT, GPT-3.

Can I fine tune GitHub Copilot model?

I have already looked into the [https://copilot.github.com/](https://copilot.github.com/) but cant find the details. Would really appreciate if someone had fine tuned Github Copilot.

---

<div class="post-metadata">

**Author:** ![lvwerra](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lvwerra/32/2427_2.png) [@lvwerra](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flvwerra)%5C
**Post date:** [June 9, 2022, 12:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-fine-tune-fine-tune-github-copilot%2F18889%2F2 "2022-06-09T12:34:10Z")

</div>

Hi @neo-benjamin

The Codex model that’s powering the Copilot product is not open sourced. However, there are a few models similar to Codex available on the Hugging Face Hub such as Incoder or CodeGen:

> **[facebook/incoder-6B · Hugging Face](https://huggingface.co/facebook/incoder-6B)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

> **[Salesforce/codegen-16B-multi · Hugging Face](https://huggingface.co/Salesforce/codegen-16B-multi)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![neo-benjamin](https://avatars.discourse-cdn.com/v4/letter/n/f6c823/32.png) [@neo-benjamin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fneo-benjamin)%5C
**Post date:** [June 9, 2022, 8:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-fine-tune-fine-tune-github-copilot%2F18889%2F3 "2022-06-09T20:19:57Z")

</div>

How to fine tune Codegen? Are the steps documented?

---

<div class="post-metadata">

**Author:** ![lvwerra](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lvwerra/32/2427_2.png) [@lvwerra](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flvwerra)%5C
**Post date:** [June 24, 2022, 1:40pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-fine-tune-fine-tune-github-copilot%2F18889%2F4 "2022-06-24T13:40:30Z")

</div>

You can have a look at the language modeling examples. The should work for any auto regressive model such as GPT-2 or CodeGen: [transformers/examples/pytorch/language-modeling at main · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling)