# PreTrain Wav2Vec2 in Swedish

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-wav2vec2-in-swedish%2F7604%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [June 29, 2021, 8:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-wav2vec2-in-swedish%2F7604 "2021-06-29T08:01:25Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![birgermoell](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/birgermoell/32/2052_2.png) [@birgermoell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbirgermoell)%5C
**Post date:** [June 29, 2021, 8:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-wav2vec2-in-swedish%2F7604%2F1 "2021-06-29T08:01:25Z")

</div>

# PreTrain Wav2Vec2 in Swedish

There is currently only a multilingually pretrained model for Swedish Wav2Vec2. Let’s make a Wav2Vec2 only pretrained on Swedish.

## Model

A randomly initialized Wav2Vec2 model

## Datasets

One can make use [Common Voice](https://commonvoice.mozilla.org/en/languages) the dataset is also available through the `datasets` library here: [common\_voice · Datasets at Hugging Face](https://huggingface.co/datasets/common_voice).

## Available training scripts

FlaxWav2Vec2 will be merged soon: [[Flax] Add wav2vec2 by patrickvonplaten · Pull Request #12271 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/pull/12271) and a pretraining script should be relatively easy to be merged.

## (Optional) Desired project outcome

The best Swedish ASR model.

## (Optional) Challenges

It might make sense to use more data than just common voice.

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 29, 2021, 8:03am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-wav2vec2-in-swedish%2F7604%2F2 "2021-06-29T08:03:35Z")

</div>

Maybe also interesting for @marma ?

---

<div class="post-metadata">

**Author:** ![marma](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marma/32/23010_2.png) [@marma](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmarma)%5C
**Post date:** [June 29, 2021, 10:20am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-wav2vec2-in-swedish%2F7604%2F3 "2021-06-29T10:20:17Z")

</div>

It would be interesting, the problem for us is that we cannot share data. However, we have been pretraining a large Wav2Vec2 at the KB for some time and right now, in the middle of training, it (marginally) outperforms the Swedish VoxPopuli. I’ll write something up and publish it before I go on vacation in two-three days. At least the license will be better than CC NonCommercial.

 ![Screenshot 2021-06-15 at 16.01.48](https://us1.discourse-cdn.com/hellohellohello/original/2X/f/f3bb3c73e4e8d8c0c26b1805ad7aa7eaf25ad149.png)

---

<div class="post-metadata">

**Author:** ![marma](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marma/32/23010_2.png) [@marma](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmarma)%5C
**Post date:** [June 29, 2021, 10:23am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-wav2vec2-in-swedish%2F7604%2F4 "2021-06-29T10:23:25Z")

</div>

“It” being the model in this case. We’ll detail it in a pre-print at some point.