# Finetuning Whisper with prompts
**URL:**
**Author:**  [@AvivSham](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAvivSham)%5C
**Post date:** [June 13, 2023, 9:44am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetuning-whisper-with-prompts%2F43053%2F1 "2023-06-13T09:44:28Z")
Hi All,
I’m trying to finetune Whisper by resuming its pre-training task and adding initial prompts as part of the model’s forward pass. I saw this [amazing tutorial](https://huggingface.co/blog/fine-tune-whisper), however, it does not contain a section about using prompts as part of the fine-tuning dataset.
Thanks!
---
**Author:**  [@andercorral](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fandercorral)%5C
**Post date:** [June 29, 2023, 10:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetuning-whisper-with-prompts%2F43053%2F2 "2023-06-29T10:46:02Z")
Any news on this?
---
**Author:**  [@andercorral](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fandercorral)%5C
**Post date:** [July 11, 2023, 9:12am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetuning-whisper-with-prompts%2F43053%2F3 "2023-07-11T09:12:31Z")
Hi @AvivSham,
I started digging into the actual code and I just realized that the Whisper tokenizer can accept two sentences as input just as models such as BERT do. For BERT-like models the two input sentences are concated and separated by a `[SEP]` token:
`[CLS] sentence1 [SEP] sentence2 [SEP]`
This behaviour is kept in the Whisper tokenizer too for API consistency issues, although it is not actually used during the finetuning process. In the current code, it simply concatenates both sentences if passed.
>
In order to avoid too many changes in the code, I would simply replace that line with the following code, so that it matches the format stated in the original paper:
```auto
start_of_prev_id = self.all_special_ids[-3]
return [start_of_prev_id] + token_ids_1 + self.prefix_tokens + token_ids_0 + [self.eos_token_id]
```
After that change, passing both the actual transciption and the prompt to the tokenizer it should return the expected format:
```python
processor = WhisperProcessor.from_pretrained("openai/whisper-small", language="english", task="transcribe")
processor.tokenizer(" and the actual transcription is this.", "This is the prompt")
# The code above should return this
# <|startofprev|>This is the prompt<|startoftranscript|> <|transcribe|> <|en|> <|notimestamps|> and the actual transcription is this.<|endoftext|>
```
I hope this serves as the starting point to finetune Whisper with prompts.
---
**Author:**  [@kenfus](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkenfus)%5C
**Post date:** [January 16, 2024, 9:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetuning-whisper-with-prompts%2F43053%2F4 "2024-01-16T21:17:46Z")
Hello! Thanks for your idea.
Now you have the correct IDs. However, how do you now pass this to the Trainer? I thought that is probably handled automatically but the more I go into the code, the more I am confused. With this, we need to split this up between, so that the Model (in your example):
- generates with prompt\_id, which are up until and without `<|startoftranscript|>`
- decoder\_input\_ids with and up until `<|notimestamps|>`.
When I started to look into the trainer, which I know generates decoder\_input\_ids from the labels, I found this:
> ```
> if labels is not None:
>
> if decoder_input_ids is None and decoder_inputs_embeds is None:
> decoder_input_ids = shift_tokens_right(
> labels, self.config.pad_token_id, self.config.decoder_start_token_id
> )
>
> ```
> ```
> def shift_tokens_right(input_ids: torch.Tensor, pad_token_id: int, decoder_start_token_id: int):
>
> shifted_input_ids = input_ids.new_zeros(input_ids.shape)
> shifted_input_ids[:, 1:] = input_ids[:, :-1].clone()
> shifted_input_ids[:, 0] = decoder_start_token_id
>
> if pad_token_id is None:
> raise ValueError("self.model.config.pad_token_id has to be defined.")
> # replace possible -100 values in labels by `pad_token_id`
> shifted_input_ids.masked_fill_(shifted_input_ids == -100, pad_token_id)
>
> return shifted_input_ids
>
> ```
from: [transformers/src/transformers/models/whisper/modeling\_whisper.py at main · huggingface/transformers (github.com)](https://github.com/huggingface/transformers/blob/main/src/transformers/models/whisper/modeling_whisper.py#L1816)
However, this just made me more confused: Why does padding and shift tokens to the right make the correct decoder\_input\_ids? In my opinion, it should be the following (of course as IDS) and not just the labels shifted to the right with a `<|startoftranscript|>` added
> \<|startofprev|\>This is the prompt\<|startoftranscript|\> \<|transcribe|\> \<|en|\> \<|notimestamps|\>
Any input to this? Thank you very much!