# Fine-Tuning / Pre-Training Tips

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-pre-training-tips%2F15367%3E%5C
**Category:** 🤗Transformers\
**Created:** [March 4, 2022, 5:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-pre-training-tips%2F15367 "2022-03-04T17:21:23Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [March 4, 2022, 5:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-pre-training-tips%2F15367%2F1 "2022-03-04T17:21:23Z")

</div>

Copy-pasting some internal discussions here as it might be interesting for everyone. Feel free to comment / ask questions!

# Fine-tuning

I only have experience with training models \<= 2B parameters and only in PyTorch / Flax and only with using Accelerate, Examples/Trainer on TPU & GPU.

- _TPU or GPU?_ For fine-tuning models \<= 2B if someone doesn’t have experience with TPUs, it doesn’t really make sense to jump into TPUs (might be different for TF) because the models are fine-tuned pretty quickly usually (less than a day), so the actual cost that’s saved per training run is negligible. Also, it’s just much easier to quickly experiment with PyTorch then with Flax (more docs in PyTorch, much better support for GPU)
- _[Trainer](https://huggingface.co/docs/transformers/v4.16.2/en/main_classes/trainer) or [accelerate](https://huggingface.co/docs/accelerate/)?_ Then I usually use the Trainer for fine-tuning because it works out of the box for pretty much all use-cases. I haven’t used `accelerate` yet for fine-tuning only, but we also have lots of working examples for it now - so I think both Trainer and accelerate make sense here. Trainer is better atm IMO because it has great support to directly upload checkpoints to the Hub, automatically creates the README, etc…
- Optimizations:
- _[fp16](https://huggingface.co/docs/transformers/v4.16.2/en/main_classes/trainer#transformers.TrainingArguments.fp16)? bfloat16?_ Now for more or less all models that come from Facebook/Fairseq or Microsoft (BART, Wav2Vec2, Roberta, WavLM, DialoGPT) I would do training in `fp16` because these models have all been pretrained in PyTorch usually and also usually in `fp16` . Lots of time and memory can be saved for more or less the same performance. I’ve actually never **not** trained in `fp16` for Facebook or MS models. For Google models, especially T5, they have mostly been pre-trained in bfloat16 on TPU which means that fine-tuning in `fp16` might break! This is the case for all mid- to larger-sized T5 models. Then I usually just use full `fp32` precision because `bfloat16` is only very recently support in native PyTorch I think. Guess [@Stas](https://huggingface.slack.com/team/U01KAUV2W1Z) knows more here.
- _Other optimizations?_ PyTorch is not compiled during training usually so it works well with dynamic batches. This means I **always** use dynamic padding when training in PyTorch by setting _[group\_by\_length](https://huggingface.co/docs/transformers/v4.16.2/en/main_classes/trainer#transformers.TrainingArguments.group_by_length)_ to True. Group by length, groups inputs according to their length and to make sure batches contain input samples of more or less the same length. This has a couple of advantages over not using it:
- More efficient since less padding tokens are processed
- Less padding can also mean for some models (like Wav2Vec2), less instability because less tokens are masked out with `10,000.0`
- [@Sylvain Gugger](https://huggingface.slack.com/team/U0145J1FEJK) added a very nice hack when he added `group_by_length` which puts the largest batch first. This means, you’ll know directly whether your model with OOM during training and don’t have any bad surprises after 2 hours of training
- I also always use _[gradient\_checkpointing](https://huggingface.co/docs/transformers/v4.16.2/en/main_classes/trainer#transformers.TrainingArguments.gradient_checkpointing)_ because it saves **a lot** of memory during training, especially when the model has many layers \*\*If the model still doesn’t fit into memory, first I replace the `torch's` native Adam optimizer (it’s called by default) with the [8bit adam optimizer](https://github.com/facebookresearch/bitsandbytes) which did in some cases save a lot of memory for me. If this also doesn’t work I make use of [gradient\_accumulation\_steps](https://github.com/facebookresearch/bitsandbytes)

# Pre-Training

I only have experience with training models \<= 2B parameters and only in PyTorch / Flax and only with using Accelerate, Examples/Trainer on TPU & GPU.

- TPU or GPU? Here I do recommend using TPU if the person is accustomed to TPUs. It can lead to significant speed-ups and cost savings, e.g.: [https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling#runtime-evaluation](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling#runtime-evaluation) . PyTorch/XLA on TPU can work very well, but doesn’t have great support. Flax TPU is more robust on faster than PyTorch/XLA in my experiments, but requires someone to know JAX/Flax. For more model-specific libaries, the [TF-T5](https://github.com/google-research/text-to-text-transfer-transformer) an [Jax-T5](https://github.com/google-research/t5x) libraries are great to pretrain T5 and it’s easy to convert models from those libraries to Transformers.
- If I use PyTorch, then I use accelerate rather than the Trainer because I want to have a more customized training loop (log special metrics, …)
- For pre-training the same optimization tips apply then the ones written above, except for `group_by_length` which is often not necessary because one can “blockify” the data so that no padding at all is needed

# Helpful additional links

- [https://huggingface.co/docs/transformers/v4.16.2/en/performance](https://huggingface.co/docs/transformers/v4.16.2/en/performance)
- [https://huggingface.co/docs/transformers/v4.16.2/en/parallelism](https://huggingface.co/docs/transformers/v4.16.2/en/parallelism)

---

<div class="post-metadata">

**Author:** ![redagavin](https://avatars.discourse-cdn.com/v4/letter/r/51bf81/32.png) [@redagavin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fredagavin)%5C
**Post date:** [August 5, 2022, 6:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-pre-training-tips%2F15367%2F2 "2022-08-05T18:51:32Z")

</div>

For Trainer and accelerate, is there a way to use both of them together? @patrickvonplaten
