# GPT-2 in DNA data

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt-2-in-dna-data%2F22753%3E%5C
**Category:** Research\
**Created:** [September 8, 2022, 11:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt-2-in-dna-data%2F22753 "2022-09-08T11:11:42Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![mdelas](https://avatars.discourse-cdn.com/v4/letter/m/b5ac83/32.png) [@mdelas](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmdelas)%5C
**Post date:** [September 8, 2022, 11:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt-2-in-dna-data%2F22753%2F1 "2022-09-08T11:11:42Z")

</div>

Dear community,

I’m trying to build a GPT-2 transformer from scratch (without any pre-train model) with DNA sequences in order to generate DNA sequences on top of smaller ones. I am a bit stuck and I couldn’t find any repo applying this kind of decoder-transformer with a DNA background, to have some clues in what’s the best tokenization, and some other technical choices…

Does someone have any references or think that’s a good idea?

Thank you in advance!

---

<div class="post-metadata">

**Author:** ![marisming](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marisming/32/43958_2.png) [@marisming](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmarisming)%5C
**Post date:** [August 6, 2023, 2:37am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt-2-in-dna-data%2F22753%2F2 "2023-08-06T02:37:18Z")

</div>

Normally the dna sequence is segmented by k-mers method.  
For example “ATCG” is segmented into ATC, TCG by 3-mers method. The k could be 6-13.  
dnabert model just use this method.

Some method also use BPE method. For example:  
gena-llm([AIRI-Institute/gena-lm-bert-base · Hugging Face](https://huggingface.co/AIRI-Institute/gena-lm-bert-base)) ,  
dangpt2([dnagpt/human\_gpt2-v1 · Hugging Face](https://huggingface.co/dnagpt/human_gpt2-v1))

The tokenizaion example:

```auto
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained('dnagpt/human_gpt2-v1') #AIRI-Institute/gena-lm-bert-base,zhihan1996/DNABERT-2-117M
tokenizer.tokenize("GAGCACATTCGCCTGCGTGCGCACTCACACACACGTTCAAAAAGAGTCCATTCGATTCTGGCAGTAG")
#result: [G','AGCAC','ATTCGCC',....]
`
```