# DNA long sequence tokenization
**URL:**
**Author:**  [@mdelas](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmdelas)%5C
**Post date:** [March 28, 2022, 9:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdna-long-sequence-tokenization%2F16154%2F1 "2022-03-28T09:13:24Z")
Hello everyone!
I am struggling with transformers in DNA data for a supervised binary classification problem. I have very long DNA sequences (the mean is 6E7 characters) and, to be able to pass longer sequences as input to the Neural Network, I am trying to tokenize using different algorithms to work with longer sequences tokens rather than only (C, G, A, T) ones.
At the moment I am trying with [HuggingFace](https://huggingface.co/docs/transformers/tokenizer_summary) to implement BPE, WordPiece, and Unigram algorithms. However, before training those models I do have to apply a pretokenizer to my data. All of them are based into “classic” language structures like `Whitespace()` but in my case I only have a list of DNA sequences like (small chunk):
```auto
['CCAGCAGCTCGGTGCGCTTGCCGCTCCAGTCGCCCAGCAGCTCGGTGCGCTTGCCGCCCCAGTCGC']
```
My intention is to group those characters to work with bigger tokens than only 1 single character. However, when I use for example `Whitespace()` , my model does not learn…
Could you recommend me some [pre\_tokenizer](https://huggingface.co/docs/tokenizers/python/latest/components.html) for passing as input to BPE, WPiece and UNIGRAM only characters?
Also, would you recommend padding sequence before or after tokenization process?
Thank you very much
---
**Author:**  [@ratish](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fratish)%5C
**Post date:** [June 29, 2023, 6:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdna-long-sequence-tokenization%2F16154%2F2 "2023-06-29T18:17:28Z")
Hello @mdelas,
I was wondering if you got the solution to your problem?
I have a similar doubt, I am currently working with the Bacteria Genome sequences for which I need to Pre-Train a Model from Scratch.
Thanks in advance!
---
**Author:**  [@marisming](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmarisming)%5C
**Post date:** [August 6, 2023, 3:08am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdna-long-sequence-tokenization%2F16154%2F3 "2023-08-06T03:08:37Z")
BPE just combine the short segments to long segment, So the pre tokenization is not necessary.
If you want, k-mers method could be used.
For example “ATCG” is segmented into ATC, TCG by 3-mers method. The k could be 6-13.
dnabert model just use this method.
There are also some pretained BPE tokenizer in huggingface, for example:
```auto
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained('dnagpt/human_gpt2-v1') #AIRI-Institute/gena-lm-bert-base,zhihan1996/DNABERT-2-117M
tokenizer.tokenize("GAGCACATTCGCCTGCGTGCGCACTCACACACACGTTCAAAAAGAGTCCATTCGATTCTGGCAGTAG")
#result: [G','AGCAC','ATTCGCC',....]
for classification problem, here is some examples:
https://huggingface.co/spaces/dnagpt/dnabert_pretrain_v1/blob/main/dnagpt/class6.ipynb
https://github.com/AIRI-Institute/GENA_LM/blob/main/notebooks/GENA_sequence_classification_example.ipynb
```