# How to create Wav2Vec2 With Language model **URL:** **Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C **Post date:** [December 10, 2021, 3:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-wav2vec2-with-language-model%2F12703%2F1 "2021-12-10T15:49:51Z") Now that language model boosted decoding is possible for Wav2Vec2 ([https://twitter.com/PatrickPlaten/status/1468999507488788480](https://twitter.com/PatrickPlaten/status/1468999507488788480) and [patrickvonplaten/wav2vec2-large-xlsr-53-spanish-with-lm · Hugging Face](https://huggingface.co/patrickvonplaten/wav2vec2-large-xlsr-53-spanish-with-lm) it’s important to know **How can one create a Wav2Vec2 + LM repo?**. Let’s explain (hopefully this is simpler in the future): 1. Install kenlm: The best guide to build kenlm is this github gist here: [kenlm/BUILDING at master · kpu/kenlm · GitHub](https://github.com/kpu/kenlm/blob/master/BUILDING) IMO 2. Create an ngram model: This is explained quite well here: [GitHub - kpu/kenlm: KenLM: Faster and Smaller Language Model Queries](https://github.com/kpu/kenlm#estimation) I wrote a short python script that allows to quickly create a ngram from a text corpus of common voice: ```python #!/usr/bin/env python3 from datasets import load_dataset import os import argparse parser = argparse.ArgumentParser() parser.add_argument( "--language", default="polish", type=str, required=True, help="Language to run comparison on. Choose one of 'polish', 'portuguese', 'spanish' or add more to this script." ) parser.add_argument( "--path_to_ngram", type=str, required=True, help="Path to kenLM ngram" ) args = parser.parse_args() ds = load_dataset("multilingual_librispeech", f"{args.language}", split="train") with open("text.txt", "w") as f: f.write(" ".join(ds["text"])) os.system(f"./kenlm/build/bin/lmplz -o 5 {args.path_to_ngram}") ## VERY IMPORTANT!!!: # After the language model is created, one should open the file. one should add a `` # The file should have a structure which looks more or less as follows: # \data\ # ngram 1=86586 # ngram 2=546387 # ngram 3=796581 # ngram 4=843999 # ngram 5=850874 # \1-grams: # -5.7532206 0 # 0 -0.06677356 # -3.4645514 drugi -0.2088903 # ... # Now it is very important also add a token to the n-gram # so that it can be correctly loaded. You can simple copy the line: # 0 -0.06677356 # and change to . When doing this you should also inclease `ngram` by 1. # The new ngram should look as follows: # \data\ # ngram 1=86587 # ngram 2=546387 # ngram 3=796581 # ngram 4=843999 # ngram 5=850874 # \1-grams: # -5.7532206 0 # 0 -0.06677356 # 0 -0.06677356 # -3.4645514 drugi -0.2088903 # ... # Now the ngram can be correctly used with `pyctcdecode` ``` See: [Wav2Vec2\_PyCTCDecode/create\_ngram.py at main · patrickvonplaten/Wav2Vec2\_PyCTCDecode · GitHub](https://github.com/patrickvonplaten/Wav2Vec2_PyCTCDecode/blob/main/create_ngram.py) Feel free to copy those lines of code. Multi-Lingual librispeech is already a very clean text corpus. You might want to pre-process other text corpora to not include any punctuation etc… As an example this step created the Spanish 5-gram here: [kensho/5gram-spanish-kenLM · Hugging Face](https://huggingface.co/kensho/5gram-spanish-kenLM) 1. Now we should load the language model in a PyCTCBeamSearchDecoder as this is the format we need. Here one should be very careful to choose exactly the same vocabulary as the Wav2Vec2’s tokenizer vocab. At first we should pick a fine-tuned Wav2Vec2 model that we would like to add a language model to. Let’s choose: [jonatasgrosman/wav2vec2-large-xlsr-53-spanish · Hugging Face](https://huggingface.co/jonatasgrosman/wav2vec2-large-xlsr-53-spanish) Now we instantiate a `BeamSearchDecoder` and save it to a folder `wav2vec2_with_lm`. E.g. you can run this code: ```python from transformers import AutoTokenizer from pyctcdecode import build_ctcdecoder tokenizer = AutoTokenizer.from_pretrained("jonatasgrosman/wav2vec2-large-xlsr-53-spanish") vocab_dict = tokenizer.get_vocab() sorted_dict = {k: v for k, v in sorted(vocab_dict.items(), key=lambda item: item[1])} decoder = build_ctcdecoder( list(sorted_dict.keys()), args.path_to_ngram, ) decoder.save_to_dir("wav2vec2_with_lm") ``` Now we should have saved the following files in `wav2vec2_with_lm`: ```auto - language_model - attrs.json - kenLM.arpa - unigrams.txt - alphabet.json ``` That’s it! Now all you need to do is to upload this to your Wav2Vec2 model so that the directory structure looks as follows: > **[patrickvonplaten/wav2vec2-large-xlsr-53-spanish-with-lm at main](https://huggingface.co/patrickvonplaten/wav2vec2-large-xlsr-53-spanish-with-lm/tree/main)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. Now, all you need to change is your decoding to: > **[patrickvonplaten/wav2vec2-large-xlsr-53-spanish-with-lm · Hugging Face](https://huggingface.co/patrickvonplaten/wav2vec2-large-xlsr-53-spanish-with-lm#changes)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. ---
**Author:** ![boris](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/boris/32/19312_2.png) [@boris](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fboris)%5C **Post date:** [December 10, 2021, 4:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-wav2vec2-with-language-model%2F12703%2F2 "2021-12-10T16:07:39Z")
This is cool! Can you make the inference widget use the language model? ---
**Author:** ![edugp](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/edugp/32/2025_2.png) [@edugp](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fedugp)%5C **Post date:** [December 17, 2021, 6:47am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-wav2vec2-with-language-model%2F12703%2F4 "2021-12-17T06:47:20Z")
This is awesome! I have a bunch of KenLM models for different languages trained on sentence piece-tokenized Wikipedia and OSCAR here if you find them useful: [edugp/kenlm at main](https://huggingface.co/edugp/kenlm/tree/main) ---
**Author:** ![Harveenchadha](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/harveenchadha/32/3428_2.png) [@Harveenchadha](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHarveenchadha)%5C **Post date:** [December 17, 2021, 6:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-wav2vec2-with-language-model%2F12703%2F5 "2021-12-17T18:32:40Z")
Hi @patrickvonplaten Were you able to install kenlm in spaces? ---
**Author:** ![csikasote](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/csikasote/32/3487_2.png) [@csikasote](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcsikasote)%5C **Post date:** [December 18, 2021, 9:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-wav2vec2-with-language-model%2F12703%2F6 "2021-12-18T09:46:41Z")
This is awesome. Thank you! ---
**Author:** ![edugp](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/edugp/32/2025_2.png) [@edugp](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fedugp)%5C **Post date:** [December 20, 2021, 2:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-wav2vec2-with-language-model%2F12703%2F7 "2021-12-20T14:59:17Z")
@Harveenchadha you can install it adding this line to your requirements.txt: `https://github.com/kpu/kenlm/archive/master.zip` ---
**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C **Post date:** [December 22, 2021, 1:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-wav2vec2-with-language-model%2F12703%2F8 "2021-12-22T13:51:01Z")
> [@boris](#): > > inference Yes working on it! 🙂 ---
**Author:** ![RASMUS](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/rasmus/32/21273_2.png) [@RASMUS](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRASMUS)%5C **Post date:** [January 2, 2022, 6:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-wav2vec2-with-language-model%2F12703%2F10 "2022-01-02T18:16:26Z")
If someone is trying this with LM like me and has problems with AutoProcessor / AutoModelForCTC install transformers from @patrickvonplaten commit [[AutoProcessor] Correct AutoProcessor and automatically add processor… · huggingface/transformers@a139288 · GitHub](https://github.com/huggingface/transformers/commit/a1392883cedd0398a6ae39ccac7432ba72ad571b) At least I was able to load my model ( [RASMUS/wav2vec2-xlsr · Hugging Face](https://huggingface.co/RASMUS/wav2vec2-xlsr) ) with LM after that fix ---
**Author:** ![eli4s](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/eli4s/32/56317_2.png) [@eli4s](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Feli4s)%5C **Post date:** [January 7, 2022, 11:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-wav2vec2-with-language-model%2F12703%2F11 "2022-01-07T11:30:33Z")
Hi, Thank you for sharing resources ! I have come across the paper of wav2vec2, and according to the paper it seems like using a Transformer as a language model yields the best results in terms of WER. However, I do not find any other reference/resource on the subject, nor did I find the weights of the Transformer used for decoding. Is it possible that you might have any resource on the subject? (where I could find the weights of such transformer decoder) Thanks a lot for helping the community 🤗 ---
**Author:** ![yaswanth](https://avatars.discourse-cdn.com/v4/letter/y/94ad74/32.png) [@yaswanth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fyaswanth)%5C **Post date:** [January 27, 2022, 4:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-wav2vec2-with-language-model%2F12703%2F12 "2022-01-27T16:58:48Z")
> [@patrickvonplaten](#): > > `decoder.save_to_dir("wav2vec2_with_lm")` While predicting the text with Model with LM , i am getting below error. ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/8/8103770e8734e28960fc1fa91e138d706fd51c07.png) ---
**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C **Post date:** [January 29, 2022, 2:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-wav2vec2-with-language-model%2F12703%2F13 "2022-01-29T14:06:16Z")
Hey @yaswanth, Sorry to reply so late. Could you add a fully reproducible code sample? This is as minimal as possible? 🙂 ---
**Author:** ![yaswanth](https://avatars.discourse-cdn.com/v4/letter/y/94ad74/32.png) [@yaswanth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fyaswanth)%5C **Post date:** [January 29, 2022, 2:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-wav2vec2-with-language-model%2F12703%2F14 "2022-01-29T14:58:41Z")
code: ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/2/2604cba9513a8b35ea52f06b1a39e8e25f7cb136.png) Model\_Files: ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/7/7c928521b71a2090d2d178b522afe11f0920d978.png) Error: ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/7/72ef267e8b43bd96bce3d7b4c0e09b9ff2049e12.png) ---
**Author:** ![diegoseto](https://avatars.discourse-cdn.com/v4/letter/d/6a8cbe/32.png) [@diegoseto](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdiegoseto)%5C **Post date:** [May 10, 2022, 4:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-wav2vec2-with-language-model%2F12703%2F17 "2022-05-10T16:13:15Z")
Hi @patrickvonplaten @philschmid Is it possible to deploy Wav2Vec2 with a KenLM Language Model in Amazon SageMaker? I was following this article **[https://www.philschmid.de/automatic-speech-recognition-sagemaker](https://www.philschmid.de/automatic-speech-recognition-sagemaker)** but i couldn’t find an option in the sagemaker HF library to use a language model. ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fphilschmid)%5C **Post date:** [May 10, 2022, 5:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-wav2vec2-with-language-model%2F12703%2F18 "2022-05-10T17:10:01Z")
Hey @diegoseto, Yes, it is possible by deploying your model from Amazon S3 with a custom `inference.py` including the inference code. Here is an [example](https://github.com/huggingface/notebooks/blob/main/sagemaker/17_custom_inference_script/sagemaker-notebook.ipynb) notebook for using a `sentence-transformers`. You just need to modify the `inference.py` and how the `model.tar.gz` is created. > ---
**Author:** ![diegoseto](https://avatars.discourse-cdn.com/v4/letter/d/6a8cbe/32.png) [@diegoseto](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdiegoseto)%5C **Post date:** [May 10, 2022, 5:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-wav2vec2-with-language-model%2F12703%2F19 "2022-05-10T17:45:34Z")
Cool. Thank you, i will try this. Thanks for the quick reply too 😃 ---
**Author:** ![jrf](https://avatars.discourse-cdn.com/v4/letter/j/3be4f8/32.png) [@jrf](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjrf)%5C **Post date:** [May 5, 2023, 6:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-wav2vec2-with-language-model%2F12703%2F20 "2023-05-05T18:17:34Z")
Hi everyone - I’m pretty sure I know the answer to this but just making sure. Has anyone managed to use the processor with a kenlm language model on Windows? I created my .arpa file on a Linux os hoping to be able to use it with wav2vec2 on Windows without actually building/installing kenlm, but this does not appear to be the case. Even when I create the processor, bundled with the lm, on Linux and then load the processor from\_pretrained on my Windows script (based on the blog), it still wants to call kenlm and crashes. Just want to confirm that there is no way around this and that, at this time, Linux is mandatory for using wav2vec2 with a language model, even with an already existing .arpa file that has been created somewhere else. thanks! Jonathan