# Pretrain Wav2vec2 in Russian

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-wav2vec2-in-russian%2F7784%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [June 30, 2021, 10:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-wav2vec2-in-russian%2F7784 "2021-06-30T22:51:07Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![gorodecki](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@gorodecki](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgorodecki)%5C
**Post date:** [June 30, 2021, 10:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-wav2vec2-in-russian%2F7784%2F1 "2021-06-30T22:51:07Z")

</div>

# Pretrain Wav2vec2 in Russian

There is currently only a multilingually pretrained model for Russian Wav2Vec2. Let’s make a Wav2Vec2 only pretrained on Russian.

## 2. Language

The model will be trained in Russian.

## 3. Model

A randomly initialized Wav2Vec2 model

## 4. Datasets

Part of CommonVoice - [CommonVoice](https://commonvoice.mozilla.org/ru/datasets) - 35h  
SberGolos - [Golos](https://github.com/sberdevices/golos) - 1095h  
SovaDataset - [SovaDataset](https://github.com/sovaai/sova-dataset) - 400h

## 5. Training scripts

We can make use of [run\_wav2vec2\_pretrain\_flax](https://github.com/huggingface/transformers/blob/master/examples/research_projects/jax-projects/wav2vec2/run_wav2vec2_pretrain_flax.py) to train the model.

## 6. (Optional) Challenges

Dataset have many hours, training will take a long time

## 7. (Optional) Desired project outcome

The best Russian ASR model.

---

<div class="post-metadata">

**Author:** ![gorodecki](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@gorodecki](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgorodecki)%5C
**Post date:** [June 30, 2021, 10:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-wav2vec2-in-russian%2F7784%2F2 "2021-06-30T22:52:07Z")

</div>

@anton-l 🕶 😇 :jax:

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [July 1, 2021, 9:57am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-wav2vec2-in-russian%2F7784%2F3 "2021-07-01T09:57:04Z")

</div>

Would be awesome to see this happen!