# German ASR: Fine-Tuning Wav2Vec2
**URL:**
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [March 18, 2021, 7:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgerman-asr-fine-tuning-wav2vec2%2F4558%2F1 "2021-03-18T07:46:40Z")
Hey everybody,
I’m planning on fine-tuning XLSR-Wav2Vec2 in German. I’m happy about any tips / collaborations with the community 🙂
---
**Author:**  [@burrjh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fburrjh)%5C
**Post date:** [March 18, 2021, 8:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgerman-asr-fine-tuning-wav2vec2%2F4558%2F2 "2021-03-18T08:53:34Z")
Hi Patrick!
I will work on the german dataset, too, and would very much appreciate discussions and collaborations, because - to be honest - I have never done it before.
---
**Author:**  [@stefan-it](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fstefan-it)%5C
**Post date:** [March 18, 2021, 9:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgerman-asr-fine-tuning-wav2vec2%2F4558%2F3 "2021-03-18T09:40:29Z")
Hi Patrick,
German dataset is very resource intensive, one step before actual fine-tuning/training the complete dataset folder has a size of 830GB. However, 64GB RAM seems to be sufficient for the whole pre-processing stuff.
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [March 18, 2021, 12:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgerman-asr-fine-tuning-wav2vec2%2F4558%2F4 "2021-03-18T12:26:45Z")
That’s a very good point @stefan-it ! We might have to lazy load the dataset into memory then for each batch
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [March 18, 2021, 12:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgerman-asr-fine-tuning-wav2vec2%2F4558%2F5 "2021-03-18T12:27:56Z")
I’ll try to adapt the colab slighly to show how such a use case could work!
---
**Author:**  [@stefan-it](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fstefan-it)%5C
**Post date:** [March 18, 2021, 6:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgerman-asr-fine-tuning-wav2vec2%2F4558%2F6 "2021-03-18T18:29:56Z")
@patrickvonplaten would be awesome. I just found out that the
```
import librosa
import numpy as np
def resample(batch):
batch["speech"] = librosa.resample(np.asarray(batch["speech"]), 48_000, 16_000)
batch["sampling_rate"] = 16_000
return batch
common_voice_train = common_voice_train.map(resample, num_proc=1)
```
was not really working. It takes very (!) long even after 8 - 10 hours it hasn’t finished (and I’m using a very fast CPU).
I could test a fix - and fortunately I’ve found a 8TB HDD under the desk 😅
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [March 18, 2021, 8:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgerman-asr-fine-tuning-wav2vec2%2F4558%2F7 "2021-03-18T20:53:21Z")
Yes, the resampling takes a lot of time! I think `torchaudio.resample(...)` works faster actually. You might want to use this script by @valhalla instead: [transformers/run\_common\_voice.py at dcebe254fadfe142b6f0d6301cc8a875dca7d603 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/dcebe254fadfe142b6f0d6301cc8a875dca7d603/examples/research_projects/wav2vec2/run_common_voice.py#L393)
Luckily we only need to do this step once. Afterward, we can save the dataset with: [Main classes — datasets 1.5.0 documentation](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=save%20dataset#datasets.Dataset.save_to_disk)
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [March 19, 2021, 9:21am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgerman-asr-fine-tuning-wav2vec2%2F4558%2F8 "2021-03-19T09:21:06Z")
> [@patrickvonplaten](#):
>
> need
Re: lazy data loading. After a talk with @valhalla, there is actually no need for special code to run “high-resource” language models. The `datasets` library never loads the whole dataset into RAM, when applying the `.map()` function. It only loads `writer_batch_size` samples into RAM when using `.map()` - see docs [here](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=writer_batch_size#datasets.Dataset.map) and then saves the mapped batch to the disk. You can increase or decrease the function argument `writer_batch_size` when using `.map(...)` to fit your needs best.
This means that every `.map(...)` call saves a significant amount of data onto disk meaning if you use `.map(...)` three times of a dataset of size 100GB it will cache 300GB of data.
Therefore you can do two things to reduce the required amount of hard drive storage:
1. Remove the cache regularly. This can be as easy as `rm -r ~/.cache/huggingface/datasets` to remove all cached datasets or making use of this convenient function: [https://huggingface.co/docs/datasets/package\_reference/main\_classes.html?highlight=cache#datasets.Dataset.cleanup\_cache\_files](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=cache#datasets.Dataset.cleanup_cache_files) that only remove a dateset specific cache
2. Try to use as few `.map(...)` operations as possible.
---
**Author:**  [@burrjh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fburrjh)%5C
**Post date:** [March 24, 2021, 8:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgerman-asr-fine-tuning-wav2vec2%2F4558%2F9 "2021-03-24T20:31:07Z")
Hi there,
I run into a pretty basic problem when downloading the data.
Everytime, both in Google Colab and on OVHCloud the download stops at exactly 79%.
I guess it runs out of memory? But I don’t know how to fix it.
Can anybody hint me to a solution?
---
**Author:**  [@marcel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmarcel)%5C
**Post date:** [March 24, 2021, 9:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgerman-asr-fine-tuning-wav2vec2%2F4558%2F10 "2021-03-24T21:41:02Z")
I switched to OVH for this reason…
---
**Author:**  [@marcel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmarcel)%5C
**Post date:** [March 25, 2021, 1:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgerman-asr-fine-tuning-wav2vec2%2F4558%2F11 "2021-03-25T01:34:52Z")
OVHCloud you have to add additional volumes… that is what currently works for me:
```auto
ovhai job run \
--gpu 1 \
--name hf-wav2vec-de02 \
--volume data@GRA/de:/workspace/data/de:RW:cache \
--volume output_models@GRA/de:/workspace/ouptut_models/de:RW:cache \
--volume dev@GRA:/workspace/my_dev_bucket:RW:cache \
--volume cache@GRA:/workspace/.cache:RW:cache \
databuzzword/hf-wav2vec
```
---
**Author:**  [@maxidl](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmaxidl)%5C
**Post date:** [March 25, 2021, 10:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgerman-asr-fine-tuning-wav2vec2%2F4558%2F12 "2021-03-25T10:04:57Z")
Hi everyone,
I joined your efforts today.
Regarding the large disk space consumption I found that after this step:
```
# Preprocessing the datasets.
# We need to read the aduio files as arrays and tokenize the targets.
def speech_file_to_array_fn(batch):
speech_array, sampling_rate = torchaudio.load(batch["path"])
batch["speech"] = resampler(speech_array).squeeze().numpy()
batch["sampling_rate"] = 16_000
batch["target_text"] = batch["text"]
return batch
train_dataset = train_dataset.map(
speech_file_to_array_fn,
remove_columns=train_dataset.column_names,
num_proc=data_args.preprocessing_num_workers,
)
eval_dataset = eval_dataset.map(
speech_file_to_array_fn,
remove_columns=eval_dataset.column_names,
num_proc=data_args.preprocessing_num_workers,
)
```
although the batch[‘speech’] numpy arrays are of type fp32, the arrow table train\_dataset.data reports that the values are doubles. I think this is because it is converted to a python List[float] somewhere before it is added to the arrow table, and the inferred format becomes fp64.
Thus, the uncompressed size is potentially much larger, compared to compressed mp3 or uncompressed fp32.
Additionally, this step does not seem to run for me at all when using multiple workers.
Im looking to replace this step by saving raw tensors in fp32 to disk and then using a custom dataset during training.
---
**Author:**  [@maxidl](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmaxidl)%5C
**Post date:** [March 25, 2021, 7:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgerman-asr-fine-tuning-wav2vec2%2F4558%2F13 "2021-03-25T19:31:40Z")
if anyone else faces a long delay before the training starts: disable group\_by\_length
in trainer\_pt\_utils.py line 506, this is called:
` lengths = [len(feature[self.model_input_name]) for feature in dataset]`
which iterates over the whole dataset without use of dataloader workers. this took a pretty long time in my case.
In the long run in might be worth to submit a pull request to change this behavior, or add a flag to enable getting the lengths using a dataloader?
---
**Author:**  [@marcel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmarcel)%5C
**Post date:** [March 25, 2021, 8:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgerman-asr-fine-tuning-wav2vec2%2F4558%2F14 "2021-03-25T20:28:56Z")
Anyone any success? Here is my training on 3% [marcel/wav2vec2-large-xlsr-german-demo · Hugging Face](https://huggingface.co/marcel/wav2vec2-large-xlsr-german-demo)
---
**Author:**  [@maxidl](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmaxidl)%5C
**Post date:** [March 26, 2021, 9:42am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgerman-asr-fine-tuning-wav2vec2%2F4558%2F15 "2021-03-26T09:42:13Z")
No success for me yet.
Since I already shared my repo on slack, here is a link: [GitHub - maxidl/wav2vec2](https://github.com/maxidl/wav2vec2)
Currently, I’m running the preprocessing part once again, after fixing a bug (processor is not deterministic across runs, so I save it to disk in the preprocessing step now). in ~4 hours I can hopefully start training some models.
---
**Author:**  [@maxidl](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmaxidl)%5C
**Post date:** [March 26, 2021, 8:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgerman-asr-fine-tuning-wav2vec2%2F4558%2F16 "2021-03-26T20:21:55Z")
So i was stuck the whole day because of segmentation faults until I found this issue:
[https://github.com/pytorch/pytorch/issues/54752](https://github.com/pytorch/pytorch/issues/54752)
TLDR: do not use torch.set\_num\_threads()
---
**Author:**  [@maxidl](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmaxidl)%5C
**Post date:** [March 28, 2021, 10:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgerman-asr-fine-tuning-wav2vec2%2F4558%2F17 "2021-03-28T22:13:47Z")
Since the sprint is nearing the end, I uploaded my latest model checkpoint to the Hub: [maxidl/wav2vec2-large-xlsr-german · Hugging Face](https://huggingface.co/maxidl/wav2vec2-large-xlsr-german) achieving a WER of 12.62%.
Based on the loss curves the training is not converged yet, so it might be worth going for even longer training runs. For now, I only managed to get 50k steps, taking around 30 hours on a single A100. Unfortunately, I did not have great success with distributed training.
---
**Author:**  [@Elaben](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FElaben)%5C
**Post date:** [February 18, 2022, 9:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgerman-asr-fine-tuning-wav2vec2%2F4558%2F18 "2022-02-18T09:28:31Z")
Hello everyone,
I have a similar problem:
I followed the tutorial “Fine-Tune Wav2Vec2 for English ASR with Transformers”, but replacing the TIMIT database by Librispeech.
It turns out that the following code causes the whole corpus to be put in RAM:
`timit = timit.map(prepare_dataset, remove_columns=timit.column_names["train"], num_proc=4)`
Is there any way to work around this problem?
Note that I am working on a local server with 200 GB of RAM and a GeForce RTX 3090 GPU with 24 GB.
Thanks in advance for your help