# PreTrain ProteinBERT from scratch
**URL:**
**Author:**  [@timothyman](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftimothyman)%5C
**Post date:** [June 28, 2021, 12:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-proteinbert-from-scratch%2F7519%2F1 "2021-06-28T12:33:05Z")
# PreTrain ProteinBERT from scratch
ProteinBERT is a universal deep-learning model of protein sequence and function based on the BERT architecture. The goal of this project is to pretrain the ProteinBERT model in JAX/Flax for downstream finetuning tasks like predicting protein structure, post translational modifications and/or biophysical attributes. The model itself can be seen as an extension to the Protein Transformer models trained by the [Rostlab](https://huggingface.co/Rostlab).
## 2. Language
The model will be trained on protein sequences and gene ontology (GO) annotations.
## 3. Model
The classic Transformer/BERT architecture, with some additions.
## 4. Datasets
ProteinBERT is pretrained on a dataset derived from UniRef90 which consists of ~106M protein sequences. Protein Gene Ontology (GO) annotations are used as additional inputs (to help the model infer about the function of the input protein and update its internal representations and outputs accordingly).
Possible links to publicly available datasets include:
- [GO annotations from CAFA](https://www.biofunctionprediction.org/cafa-targets/cafa4ontologies.zip)
- UniRef90 XML
- UniRef90 FASTA
## 5. Training scripts
Data preprocessing scripts are available on the [Official Github Repo](https://github.com/nadavbra/protein_bert). The training script can be adapted from [run\_mlm\_flax.py](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_mlm_flax.py).
## 6. Challenges
- The data of protein sequences and GO annotations require ~1 TB of scratch disk space.
- The original paper states a pretraining time of ~28 days on a single GPU (Nvidia Quadro RTX 5000).
- Besides NLP skills there may be some bioinformatics skills required
## 7. Desired project outcome
The model that can be further finetuned to predict protein structure, post translational modifications, and/or biophysical attributes.
## 8. Reads
The following links can be useful to better understand the project and
what has previously been done.
- [ProteinBERT article](https://www.biorxiv.org/content/10.1101/2021.05.24.445464v1)
- [Official Github Repo](https://github.com/nadavbra/protein_bert)
- [Work in progress porting ProteinBERT to JAX by Saurav Maheshkar](https://github.com/SauravMaheshkar/ProteinBERT)
---
**Author:**  [@SauravMaheshkar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSauravMaheshkar)%5C
**Post date:** [June 28, 2021, 1:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-proteinbert-from-scratch%2F7519%2F3 "2021-06-28T13:36:59Z")
Thanks for the shoutout ๐
. Iโm definitely interested. Whatโs the course of action @timothyman ??
---
**Author:**  [@timothyman](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftimothyman)%5C
**Post date:** [June 28, 2021, 1:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-proteinbert-from-scratch%2F7519%2F4 "2021-06-28T13:51:57Z")
I believe we have to wait till this project is officially accepted, so that we get access to those TPUs. ๐
But I will have a look and see whether I can preprocess the training data, since that seems to be taking some time too.
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 29, 2021, 2:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-proteinbert-from-scratch%2F7519%2F5 "2021-06-29T14:11:46Z")
Yes definitely a great idea! Finalizing this project ๐
---
**Author:**  [@SauravMaheshkar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSauravMaheshkar)%5C
**Post date:** [June 30, 2021, 6:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-proteinbert-from-scratch%2F7519%2F6 "2021-06-30T06:19:15Z")
> [@SinghShreya05](#):
>
> have previously studied and explored the ProteinBERT architecture and definitely want to be a part of it
Hey join us on the [discord channel](https://discord.gg/8XgKfmZz) weโre in the #proteinbert Text Channel
---
**Author:**  [@Zakia](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FZakia)%5C
**Post date:** [July 6, 2022, 6:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-proteinbert-from-scratch%2F7519%2F7 "2022-07-06T18:34:13Z")
Hi
Can this be used for classifying for any protein-based task? Also, can the input data be in FASTA file format, or what is the best practices for input data for training, and testing?
Thank you,
Zakia