# BigBirDNA - Pretraining BigBird on DNA sequences
**URL:**
**Author:**  [@kees](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkees)%5C
**Post date:** [June 24, 2021, 11:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbigbirdna-pretraining-bigbird-on-dna-sequences%2F7315%2F1 "2021-06-24T23:43:26Z")
# BigBirDNA
Pretraining BigBird on DNA sequences. This provides a base model for downstream DNA sequence analysis tasks
## 2. Language
The model will be trained in DNA
## 3. Model
BigBird.
## 4. Datasets
All the available DNA sequences.
Possible links to publicly available datasets include:
- [GenBank Overview](http://www.ncbi.nlm.nih.gov/genbank/)
- Others can be found on [www.insdc.org](http://www.insdc.org)
## 5. Training scripts
- There is a Flax BigBird implementation at [https://github.com/huggingface/transformers/blob/master/src/transformers/models/big\_bird/modeling\_flax\_big\_bird.py](https://github.com/huggingface/transformers/blob/master/src/transformers/models/big_bird/modeling_flax_big_bird.py)
- There is [https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run\_mlm\_flax.py](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_mlm_flax.py) for word masking which we may be able to tweak for bigbird
- There is [GitHub - jerryji1993/DNABERT: DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome](https://github.com/jerryji1993/DNABERT)
that can be used to copy/tweak downstream tasks
## 6. (Optional) Challenges
- There will be quite a bit of preprocessing involved
- There will be quite a lot of data involved (about half a TB compressed)
- Besides NLP skills there may be some bioinformatics skills required
## 7. (Optional) Desired project outcome
- Pretrain a transformer that can emulate or improve on the downstream tasks mentioned reads below
## 8. (Optional) Reads
The following links can be useful to better understand the project and
what has previously been done.
- [https://arxiv.org/pdf/2007.14062.pdf](https://arxiv.org/pdf/2007.14062.pdf)
- [https://www.biorxiv.org/content/10.1101/2020.09.17.301879v1.full](https://www.biorxiv.org/content/10.1101/2020.09.17.301879v1.full)
---
**Author:**  [@SauravMaheshkar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSauravMaheshkar)%5C
**Post date:** [June 25, 2021, 11:16am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbigbirdna-pretraining-bigbird-on-dna-sequences%2F7315%2F2 "2021-06-25T11:16:06Z")
> [@kees](#):
>
> Pretraining RoBERTA on DNA sequences. This provides a base model for downstream DNA sequence analysis tasks
Sounds interesting @kees, I was infact working on something similar → a jax/flax implementation of [ProteinBERT](https://www.biorxiv.org/content/10.1101/2021.05.24.445464v1). Pretraining RoBERTa seems like a good strategy. I’m definitely interested.
---
**Author:**  [@agemagician](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fagemagician)%5C
**Post date:** [June 25, 2021, 11:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbigbirdna-pretraining-bigbird-on-dna-sequences%2F7315%2F3 "2021-06-25T11:38:26Z")
DNA Sequences are very long, it is better to use “BigBird” model than “RoBERTA”.
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 25, 2021, 5:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbigbirdna-pretraining-bigbird-on-dna-sequences%2F7315%2F4 "2021-06-25T17:28:54Z")
Agree with @agemagician - this could be a very interesting project to use BigBird for!
We have FlaxBigBird: [transformers/modeling\_flax\_big\_bird.py at master · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/master/src/transformers/models/big_bird/modeling_flax_big_bird.py) merged and the official BigBird paper also did some experiments on protein sequence modeling: [[2007.14062] Big Bird: Transformers for Longer Sequences](https://arxiv.org/abs/2007.14062)
I think one should be able to slightly tweak `https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_mlm_flax.py` for it to work well with BigBird 🙂 Also pinging our “BigBird” expert @vasudevgupta 😉
---
**Author:**  [@vasudevgupta](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvasudevgupta)%5C
**Post date:** [June 25, 2021, 5:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbigbirdna-pretraining-bigbird-on-dna-sequences%2F7315%2F5 "2021-06-25T17:44:31Z")
Thanks @patrickvonplaten for pinging me here. Training bigbird on dna-sequences sounds so interesting (Always wanted to do that 🤩🤩). I would also be happy to work on this one.
---
**Author:**  [@kees](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkees)%5C
**Post date:** [June 25, 2021, 9:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbigbirdna-pretraining-bigbird-on-dna-sequences%2F7315%2F6 "2021-06-25T21:15:33Z")
I had a look at the bigbird paper and it sounds like a great starting point - thanks for the suggestion!
---
**Author:**  [@Dimitre](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDimitre)%5C
**Post date:** [June 27, 2021, 12:41am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbigbirdna-pretraining-bigbird-on-dna-sequences%2F7315%2F7 "2021-06-27T00:41:51Z")
Hi everyone, this project seems very interesting and I would love to join, if you wanna know a little more about my background check out my [GitHub](https://github.com/dimitreOliveira), I have worked on a somewhat [similar problem](https://github.com/dimitreOliveira/COVID-19-Vaccine-Degradation-Prediction).
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [June 28, 2021, 5:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbigbirdna-pretraining-bigbird-on-dna-sequences%2F7315%2F8 "2021-06-28T17:11:11Z")
Let’s officially define this project 🙂
Putting everybody in the official sheet [here](https://docs.google.com/spreadsheets/d/1GpHebL7qrwJOc9olTpIPgjf8vOS0jNb6zR_B8x_Jtik/edit?usp=sharing). More people can still join! Leave a comment here or on the sheet if you want to change something.
---
**Author:**  [@SauravMaheshkar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSauravMaheshkar)%5C
**Post date:** [June 28, 2021, 5:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbigbirdna-pretraining-bigbird-on-dna-sequences%2F7315%2F9 "2021-06-28T17:28:21Z")
Hey @valhalla, I’m still not sure if I want to be a part of this project or the [ProteinBERT](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-proteinbert-from-scratch%2F7519) project. I’d rather be a part of ProteinBERT, incase it gets accepted
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 29, 2021, 2:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbigbirdna-pretraining-bigbird-on-dna-sequences%2F7315%2F10 "2021-06-29T14:13:06Z")
Protein BERT is accepted - I’ve added you there 🙂 Will remove you from BigBirDNA then 🙂
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 29, 2021, 2:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbigbirdna-pretraining-bigbird-on-dna-sequences%2F7315%2F11 "2021-06-29T14:13:47Z")
Let me know if you want to take part in both projects @SauravMaheshkar
---
**Author:**  [@shpotes](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fshpotes)%5C
**Post date:** [June 29, 2021, 9:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbigbirdna-pretraining-bigbird-on-dna-sequences%2F7315%2F12 "2021-06-29T21:04:42Z")
> [@agemagician](#):
>
> it is better to use “BigBird” model than “RoBERTA”
Do the attention patterns that Big Bird uses have any biological sense or could they deteriorate performance for some tasks?
I mean, BigBird induces sparsity to the model using different attention patterns (see figure below) that have a certain linguistic sense, however, I don’t know if they can be easily extrapolated to relationships between k-mers.
---
**Author:**  [@SauravMaheshkar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSauravMaheshkar)%5C
**Post date:** [June 30, 2021, 2:23am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbigbirdna-pretraining-bigbird-on-dna-sequences%2F7315%2F13 "2021-06-30T02:23:58Z")
Thanks @patrickvonplaten, I didn’t know if one could participate in two projects. I’ve done some work on ProteinBERT already, so working on BigBirDNA would be a learning opportunity. Although I’m not sure how much time I’ll be able to give to two projects. For now I’ll stick to **ProteinBERT**.
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 30, 2021, 12:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbigbirdna-pretraining-bigbird-on-dna-sequences%2F7315%2F14 "2021-06-30T12:28:01Z")
Sounds good!
---
**Author:**  [@agemagician](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fagemagician)%5C
**Post date:** [July 1, 2021, 7:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbigbirdna-pretraining-bigbird-on-dna-sequences%2F7315%2F15 "2021-07-01T19:18:43Z")
I didn’t train and compare both of them for DNA sequences.
However, the main problem that DNA sequences are extremely long, and the only option to train them is using an efficient transformer.
Currently, AFAIK, the only efficient transformer that was ported to Trax is BigBird.
---
**Author:**  [@SauravMaheshkar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSauravMaheshkar)%5C
**Post date:** [July 7, 2021, 8:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbigbirdna-pretraining-bigbird-on-dna-sequences%2F7315%2F16 "2021-07-07T20:07:21Z")
Hey @patrickvonplaten, I’m almost done with my set of contributions to the ProteinBERT project. My work was mostly focused on creating the model architecture, and I’ve almost completed the work. I was wondering if I could join the BigBird team as well. I know it’s kinda late but if it’s okay with you guys and the members of the group, I’d love to contribute to the **BigBirdDNA** project as well.
---
**Author:**  [@kees](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkees)%5C
**Post date:** [July 8, 2021, 5:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbigbirdna-pretraining-bigbird-on-dna-sequences%2F7315%2F17 "2021-07-08T05:09:48Z")
It would be awesome if you can jump in - I am a little bit in over my head with other tasks at the moment so the project has not booked a lot of progress just yet
---
**Author:**  [@GrimSqueaker](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGrimSqueaker)%5C
**Post date:** [February 13, 2022, 9:14am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbigbirdna-pretraining-bigbird-on-dna-sequences%2F7315%2F18 "2022-02-13T09:14:47Z")
Hi, one of the authors of ProteinBert here - please let me or Nadav know if you have questions for the project 🙂
---
**Author:**  [@jel4h](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjel4h)%5C
**Post date:** [May 19, 2022, 7:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbigbirdna-pretraining-bigbird-on-dna-sequences%2F7315%2F19 "2022-05-19T19:45:49Z")
Hi. I’d love to use this model when it’s ready. Do you have an ETA on it’s availability?
Thanks,
Julie
---
**Author:**  [@exnx](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fexnx)%5C
**Post date:** [February 24, 2023, 9:14am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbigbirdna-pretraining-bigbird-on-dna-sequences%2F7315%2F20 "2023-02-24T09:14:17Z")
Curious what the status of this effort is 🙂
[Next page](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbigbirdna-pretraining-bigbird-on-dna-sequences%2F7315.md%3Fpage%3D2)