# Is native Pytorch training loop much slower than Trainer?
**URL:**
**Author:**  [@giusmatera](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgiusmatera)%5C
**Post date:** [August 17, 2024, 8:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-native-pytorch-training-loop-much-slower-than-trainer%2F102791%2F1 "2024-08-17T20:56:12Z")
Hi all,
I am working on a text classification task with a “distilbert-base-uncased” checkpoint and the dataset “emotion”. When I finetune the model, I average 0.34s/it when using the HF function `Trainer` but when I use the native Pytorch training I get 29.16s/it. What am I doing wrong? Below are the two snippets, the bulk of the code is taken from [Fine-tune a pretrained model](https://huggingface.co/docs/transformers/en/training).
```auto
training_args = TrainingArguments(
output_dir=model_name,
num_train_epochs=2,
learning_rate=2e-5,
per_device_train_batch_size=64,
per_device_eval_batch_size=64,
weight_decay=0.01,
evaluation_strategy="epoch",
disable_tqdm=False,
log_level="error",
)
trainer = Trainer(
model=model,
args=training_args,
compute_metrics=compute_metrics,
train_dataset=emotions_encoded["train"],
eval_dataset=emotions_encoded["validation"],
tokenizer=tokenizer,
)
trainer.train()
```
```auto
model.train()
for epoch in range(num_epochs):
for batch in train_dataloader:
batch = {k: v.to(device) for k, v in batch.items()}
outputs = model(**batch)
loss = outputs.loss
loss.backward()
optimizer.step()
lr_scheduler.step()
optimizer.zero_grad()
progress_bar.update(1)
```
---
**Author:**  [@bariskurtkaya](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbariskurtkaya)%5C
**Post date:** [August 18, 2024, 7:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-native-pytorch-training-loop-much-slower-than-trainer%2F102791%2F2 "2024-08-18T07:38:28Z")
Hi @giusmatera!
Commonly, the last layer of the model will be randomly initialized and trained in fine-tuning process. Therefore, there is no need to calculate previous gradients for the fine-tuning process. However, in the PyTorch code, if you didn’t set requires\_grad=False for the other layers, it is probably training the whole architecture instead of just the last layer. I believe that is the main reason.
Additionally, I couldn’t find any comparisons between HuggingFace Trainer and PyTorch Trainer. So, I am not sure of the answer to that question.
---
**Author:**  [@giusmatera](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgiusmatera)%5C
**Post date:** [August 18, 2024, 11:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-native-pytorch-training-loop-much-slower-than-trainer%2F102791%2F3 "2024-08-18T11:04:58Z")
Hi @bariskurtkaya! Thanks so much for your reply!
I suspect (but I might be wrong) that `Trainer()` fine-tunes per se all the weights of the pre-trained model. This can be checked using the example from [Fine-tune a pretrained model](https://huggingface.co/docs/transformers/training#trainer), with `[param for param in model.parameters()]` right before and right after the call of `Trainer()`. Could the author of the tutorial - @sgugger - confirm?
My idea was that something in the implementation of the native Pytorch training loop was off, but I was not able to understand what. The only reference I found online is [Huggingface Transformers (PyTorch) - Custom training loop doubles speed? - Stack Overflow](https://stackoverflow.com/questions/75952444/huggingface-transformers-pytorch-custom-training-loop-doubles-speed), where they face the opposite problem i.e., native Pytorch training loop performs better than Trainer.
Could anyone help?
Best,
Giuseppe
---
**Author:**  [@OscarNav](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FOscarNav)%5C
**Post date:** [November 3, 2024, 4:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-native-pytorch-training-loop-much-slower-than-trainer%2F102791%2F4 "2024-11-03T16:24:34Z")
Hi @giusmatera.
Did you solve the problem? I’m having a similar issue but with GPT-2.
---
**Author:**  [@DylanAndrew](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDylanAndrew)%5C
**Post date:** [November 11, 2024, 7:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fis-native-pytorch-training-loop-much-slower-than-trainer%2F102791%2F5 "2024-11-11T07:24:42Z")
You’re spot on! If `requires_ grad` isn’t set to `False` for earlier layers, Py Torch ends up training the whole model instead of just the last layer. Freezing the earlier layers by setting requires\_ grad=False` helps focus training where it’s needed.