Confusion with freezing Whisper's feature encoder

progmars
Tue, 10 Feb 2026 08:41:44 +0000

I am finetuning Whisper Large v3 Turbo using a script based on transformers/examples/pytorch/speech-recognition/run_speech_recognition_seq2seq.py at v4.57.6 · huggingface/transformers · GitHub

In general, it seems to be working fine. Using about 40h from Mozilla Common Voice dataset and running jiwer on unseen test data, I could reduce WER and WIL almost to half for Latvian language, when compared to the default non-finetuned non-Turbo Large model. (Haven’t measured timestamp accuracy - not needed for my specific case).

However, when discussing my training results with Google Gemini, it recommended to freeze feature encoder to make training more efficient and not affect the part of the model that “recognizes general human voice features“ and instead train only “the brain of word recognition“. Additionally, it suggested to reduce batch size from 64 to 32 (and half the learning rate) to teach the model “to generalize better“.

I’m usually quite skeptical about everything that AI says, but I got curious and wanted to try it out.
I see that by default the flag is assumed to be True in the training script code:

freeze_feature_encoder: bool = field(
        default=True,

However, all examples I’ve seen are setting it to false in the cmd or sh scripts. I tried setting it to True. And surprise - got AttributeError: ‘WhisperForConditionalGeneration’ object has no attribute ‘freeze_feature_encoder’ on the line:

if model_args.freeze_feature_encoder:
    model.freeze_feature_encoder()

Gemini was quite insistent that I should try it and, as a workaround, suggested the following code:

model.encoder.conv1.weight.requires_grad = False
if model.encoder.conv1.bias is not None:
    model.encoder.conv1.bias.requires_grad = False

model.encoder.conv2.weight.requires_grad = False
if model.encoder.conv2.bias is not None:
    model.encoder.conv2.bias.requires_grad = False

Still, it could not offer any credible sources of information to confirm that it makes sense to freeze the feature encoder when attempting to improve model’s language understanding. When I nagged it with questions, it seems to have latched onto the fact that the setting is default=True,and that it is generally recommended to freeze feature encoder in other types of models besides Whisper, where freeze_feature_encoder function is available.

So, what’s going on there? Why doesn’t WhisperForConditionalGeneration implement freeze_feature_encoder ? Did someone forgot it or is it actually useless idea to apply this for Whisper?