# Train Dutch FlaxBigBird
**URL:**
**Author:**  [@Dat](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDat)%5C
**Post date:** [June 24, 2021, 11:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-dutch-flaxbigbird%2F7312%2F1 "2021-06-24T23:03:21Z")
# FlaxBigBird for Dutch language
Currently, there are only a very limited amount of long-range sequence models for other languages than English on the hub. The goal of this project is to create a strong Dutch FlaxBigBird model.
## Model
A randomly initialized FlaxBigBird model.
## Datasets
[CC-100](http://data.statmt.org/cc-100/)
[mC4](https://github.com/allenai/allennlp/discussions/5265)
## Available training scripts
Scripts to pre-train Flax models are here: [https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)
Scripts for fine-tuning and evaluation for FlaxBigBird just got released here: [https://github.com/huggingface/transformers/tree/master/examples/research\_projects/jax-projects/big\_bird](https://github.com/huggingface/transformers/tree/master/examples/research_projects/jax-projects/big_bird) Thanks to @vasudevgupta
## (Optional) Desired project outcome
The desired project output is a strong Dutch FlaxBigBird model in Dutch. For downstream tasks that require long sequences (e.g. long text classification).
## (Optional) Challenges
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 25, 2021, 5:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-dutch-flaxbigbird%2F7312%2F2 "2021-06-25T17:31:23Z")
Agree it would be great to have more long-range sequence models for Dutch! We don’t have Longformer implemented in Jax/Flax - maybe FlaxBigBird could be an option?
---
**Author:**  [@Dat](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDat)%5C
**Post date:** [June 25, 2021, 7:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-dutch-flaxbigbird%2F7312%2F3 "2021-06-25T19:32:09Z")
Yes even better! Thanks for the suggestion, I will change it.
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 29, 2021, 2:40pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-dutch-flaxbigbird%2F7312%2F4 "2021-06-29T14:40:32Z")
Let’s try to get some participants for this project - think it’s a great idea 🙂
---
**Author:**  [@yhavinga](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fyhavinga)%5C
**Post date:** [July 2, 2021, 8:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-dutch-flaxbigbird%2F7312%2F5 "2021-07-02T08:10:39Z")
I could help a bit here and there. I have experience with getting and preprocessing texts, comfortable in the shell and python. Knowledge of transformer architectures from a users perspective. Unfortunately missed the flax/jax intro’s.
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [July 2, 2021, 3:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-dutch-flaxbigbird%2F7312%2F6 "2021-07-02T15:26:58Z")
Wuhu, let’s create it!
---
**Author:**  [@jezebleum](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjezebleum)%5C
**Post date:** [July 2, 2021, 4:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrain-dutch-flaxbigbird%2F7312%2F7 "2021-07-02T16:08:25Z")
I’ll join! Sounds like a fun project.