# Whisper large-v3 finetuning
**URL:**
**Author:**  [@sindas](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsindas)%5C
**Post date:** [April 16, 2024, 7:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhisper-large-v3-finetuning%2F81996%2F1 "2024-04-16T07:38:51Z")
I’m trying to finetune whisper model using HuggingFace following this blog post [Fine-Tune Whisper For Multilingual ASR with 🤗 Transformers](https://huggingface.co/blog/fine-tune-whisper) and by adding Lora with approximatively 50h of annotated audio. While the finetuning works fine for the medium & large-v2 versions, when i try with the large-v3 the model starts to hallucinate after 400steps, and by hallucination I mean increasing WER with repeating words (i added prediction logs to see how the model reacts) (the training & eval loss are still decreasing). (modifié)
---
**Author:**  [@basalegend](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbasalegend)%5C
**Post date:** [June 18, 2024, 8:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhisper-large-v3-finetuning%2F81996%2F2 "2024-06-18T08:48:57Z")
Hello! I have absolutely the same problem, but hallucinations and repetitions start from different number of steps, depends on size of my own datasets. Have you found the solution?
---
**Author:**  [@cautroi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcautroi)%5C
**Post date:** [October 23, 2024, 4:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhisper-large-v3-finetuning%2F81996%2F3 "2024-10-23T16:19:38Z")
I have the same issue ☹ . Fine-tuning the lagre-v3 is very terrible ☹
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [October 23, 2024, 4:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhisper-large-v3-finetuning%2F81996%2F4 "2024-10-23T16:49:07Z")
Is this perhaps a well-known problem?
> **[openai/whisper-large-v3-turbo · Why continue the charade? to get people to...](https://huggingface.co/openai/whisper-large-v3-turbo/discussions/34#671627a2b7547165a4356210)**
>
> You can't fine-tine a whisper model. This: https://huggingface.co/blog/fine-tune-whisper is only real on paper. Test your models.. Don't rely on metrics.
---
**Author:**  [@rifasca](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frifasca)%5C
**Post date:** [April 24, 2025, 8:02am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhisper-large-v3-finetuning%2F81996%2F5 "2025-04-24T08:02:48Z")
I encountered issues while fine-tuning the Whisper-large-v3 model on a 100-hour Arabic dataset using the LoRA-PEFT approach. The resulting transcriptions were highly inaccurate, with excessive hallucinations and frequent duplication of characters.
---
**Author:**  [@YuCeong-May](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FYuCeong-May)%5C
**Post date:** [June 17, 2025, 5:03am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhisper-large-v3-finetuning%2F81996%2F6 "2025-06-17T05:03:15Z")
Hello , have you found a solution for this problem? 😇
I also have a large number of hallucinations for Japanese fine-tuning with about 100hs.
But when i use a small learning-rate , the first epoch have a good result , then it become worse after warmup steps.
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [June 17, 2025, 5:36am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhisper-large-v3-finetuning%2F81996%2F7 "2025-06-17T05:36:23Z")
Is there something wrong with the v3-large model itself…?
> **[Fine-Tune Whisper For Multilingual ASR with 🤗 Transformers](https://huggingface.co/blog/fine-tune-whisper)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
> **[Medium and Large models hallucination · openai whisper · Discussion #678](https://github.com/openai/whisper/discussions/678)**
>
> Hi, I am trying to train Whisper Medium model on my custom dataset. (Using this guide) I did training on small models and it achieves very good results with WER 10 on my custom dataset. However usi...
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [June 17, 2025, 5:39am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhisper-large-v3-finetuning%2F81996%2F8 "2025-06-17T05:39:59Z")
Someone found the cause…?
> **[Share your hallucinations here · openai whisper · Discussion #1873](https://github.com/openai/whisper/discussions/1873)**
>
> Please share below any links to audio/video files that you have found to induce hallucinations in Whisper. You may include: The audio/video file itself Timestamps where hallucinations occur (unless...
> **[Hallucination on audio with no speech · openai whisper · Discussion #1606](https://github.com/openai/whisper/discussions/1606)**
>
> Issue I am trying to transcribe from the speech recorded using mic from front-end and will be sent to back-end where the recorded audio file is transcribed using whisper API (Streaming for every 5 ...
---
**Author:**  [@greatFiras](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FgreatFiras)%5C
**Post date:** [June 21, 2025, 4:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhisper-large-v3-finetuning%2F81996%2F9 "2025-06-21T16:46:27Z")
have you found a solution to that?
---
**Author:**  [@Ridma](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRidma)%5C
**Post date:** [July 1, 2025, 7:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhisper-large-v3-finetuning%2F81996%2F10 "2025-07-01T07:54:33Z")
Hi! Can anyone share whith me their code for fine-tuning with LoRA, please?
---
**Author:**  [@Ridma](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRidma)%5C
**Post date:** [July 1, 2025, 7:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhisper-large-v3-finetuning%2F81996%2F11 "2025-07-01T07:55:07Z")
Hi! Can you share whith me your code for fine-tuning with LoRA, please?
---
**Author:**  [@mekpro](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmekpro)%5C
**Post date:** [July 25, 2025, 5:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhisper-large-v3-finetuning%2F81996%2F12 "2025-07-25T05:10:33Z")
Unsloth had LoRA finetune Whisper notebook here:
> **[Google Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Whisper.ipynb#scrollTo=LjY75GoYUCB8)**