# Trainer and Accelerate
**URL:**
**Author:**  [@Indramal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FIndramal)%5C
**Post date:** [November 20, 2022, 7:39am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-and-accelerate%2F26382%2F1 "2022-11-20T07:39:47Z")
What are the differences and if Trainer can do multiple GPU work, why need Accelerate?
Accelerate use only for custom code? (add or remove something)
---
**Author:**  [@brando](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbrando)%5C
**Post date:** [June 28, 2023, 10:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-and-accelerate%2F26382%2F2 "2023-06-28T22:19:01Z")
I assume accelerate was added later and has more features like:
```auto
"""
Accelerate is a library that enables the same PyTorch code to be run across any distributed configuration by adding just
four lines of code!
tldr; handles all from cpu-gpu(s)-multi-node-tpu-tpu + deepseed + mixprecision in one simple wrapper without complicated
calls e.g. that ddp has to do for multi gpus.
ref: my notes: https://www.evernote.com/shard/s410/sh/f1158fa5-4122-0d17-d6eb-a920461e12b6/g47Qtu6j1F58zvMnJ3fWY8v6pFFWi3I_krn5155UigRUmBzr-D8td5HaQA
"""
```
---
**Author:**  [@brando](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbrando)%5C
**Post date:** [June 28, 2023, 10:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-and-accelerate%2F26382%2F3 "2023-06-28T22:20:31Z")
related: [Trainers.train() with accelerate](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainers-train-with-accelerate%2F33209)
---
**Author:**  [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C
**Post date:** [June 28, 2023, 11:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-and-accelerate%2F26382%2F4 "2023-06-28T23:46:14Z")
> [@brando](#):
>
> I assume accelerate was added later and has more features like:
The `Trainer` now uses `accelerate` as the backbone for it (our work the last few months) so it’s "do you want raw accelerate? Or the `Trainer` API). The capabilities are the same overall 🙂
---
**Author:**  [@brando](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbrando)%5C
**Post date:** [July 12, 2023, 10:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-and-accelerate%2F26382%2F5 "2023-07-12T22:21:53Z")
just saw this. Pasting this as a ref: [Hugging Face Trainer? · Issue #144 · huggingface/accelerate · GitHub](https://github.com/huggingface/accelerate/issues/144)
---
**Author:**  [@brando](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbrando)%5C
**Post date:** [July 12, 2023, 11:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-and-accelerate%2F26382%2F6 "2023-07-12T23:27:28Z")
answer:
Since the trainer already has created an accelerator obj inside it’s own code you have to do **no code changes** except for writing your own accelerate config and calling it as :
```auto
accelerate launch --config_file {path/to/config/my_config_file.yaml} {script_name.py} {--arg1} {--arg2} ...
```
An example config is given at the end.
* * *
# Long answer
My assumption was that there would be code changes, since every other accelerate tutorial showed that e.g.,
```auto
+ from accelerate import Accelerator
from transformers import AdamW, AutoModelForSequenceClassification, get_scheduler
+ accelerator = Accelerator()
model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)
optimizer = AdamW(model.parameters(), lr=3e-5)
- device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
- model.to(device)
+ train_dataloader, eval_dataloader, model, optimizer = accelerator.prepare(
+ train_dataloader, eval_dataloader, model, optimizer
+ )
num_epochs = 3
num_training_steps = num_epochs * len(train_dataloader)
lr_scheduler = get_scheduler(
"linear",
optimizer=optimizer,
num_warmup_steps=0,
num_training_steps=num_training_steps
)
progress_bar = tqdm(range(num_training_steps))
model.train()
for epoch in range(num_epochs):
for batch in train_dataloader:
- batch = {k: v.to(device) for k, v in batch.items()}
outputs = model(**batch)
loss = outputs.loss
- loss.backward()
+ accelerator.backward(loss)
optimizer.step()
lr_scheduler.step()
optimizer.zero_grad()
progress_bar.update(1)
```
but those code changes are already inside the Trainer. Their integration is so seamless it’s unclear, or perhaps it’s just not in the tutorials so one has to look at their trainer code e.g.,
```auto
if is_accelerate_available():
from accelerate import __version__ as accelerate_version
if version.parse(accelerate_version) >= version.parse("0.16"):
from accelerate import skip_first_batches
from accelerate import Accelerator
from accelerate.uti
```
So just make an accelerate config and run it e.g.,
```auto
# -----> see this ref: https://huggingface.co/docs/accelerate/package_reference/cli#accelerate-config
# ref for fsdp to know how to change fsdp opts: https://huggingface.co/docs/accelerate/usage_guides/fsdp
# ref for accelerate to know how to change accelerate opts: https://huggingface.co/docs/accelerate/basic_tutorials/launch
# ref alpaca accelerate config: https://github.com/tatsu-lab/alpaca_farm/tree/main/examples/accelerate_configs
main_training_function: main # <- change
deepspeed_config: { }
distributed_type: FSDP
downcast_bf16: 'no'
dynamo_backend: 'NO'
# seems alpaca was based on: https://huggingface.co/docs/accelerate/usage_guides/fsdp
fsdp_config:
fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
fsdp_backward_prefetch_policy: BACKWARD_PRE
fsdp_offload_params: false
fsdp_sharding_strategy: 1
fsdp_state_dict_type: FULL_STATE_DICT
# fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer # <-change
fsdp_transformer_layer_cls_to_wrap: FalconDecoderLayer # <-change
# fsdp_min_num_params: 7e9 # e.g., suggested heuristic: num_params / num_gpus = params/gpu, multiply by precision in bytes to know GBs used
gpu_ids: null
machine_rank: 0
main_process_ip: null
main_process_port: null
megatron_lm_config: { }
#mixed_precision: 'bf16'
#mixed_precision: 'no'
num_machines: 1
num_processes: 4
rdzv_backend: static
same_network: true
tpu_name: null
tpu_zone: null
use_cpu: false
```
>
---
**Author:**  [@Du5TyCh3N](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDu5TyCh3N)%5C
**Post date:** [July 15, 2023, 8:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-and-accelerate%2F26382%2F7 "2023-07-15T20:27:09Z")
Hi can someone help me out?
I am working in Google Colab to learn how to finetune a ViT model, following this tutorial [Vision Transformers (ViT) Explained + Fine-tuning in Python - YouTube](https://www.youtube.com/watch?v=qU7wO02urYU).
Trying to define the following arguments
- training and testing dataset
- feature extractor
- model
- collate function
- evaluation metric
But I have error defining TrainingArguments and Trainer, where it says
ImportError: Using the `Trainer` with `PyTorch` requires `accelerate>=0.20.1`: Please run `pip install transformers[torch]` or `pip install accelerate -U`
Where do I need to add accelerate? And how do I incorporate it with this code?
> **[Jupyter Notebook Viewer](https://nbviewer.org/github/Du5TCh3N/learningCLIP/blob/main/Finetune_ViT_Transformers.ipynb)**
>
> Check out this Jupyter notebook!
---
**Author:**  [@didi-ai](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdidi-ai)%5C
**Post date:** [July 26, 2023, 9:42am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-and-accelerate%2F26382%2F8 "2023-07-26T09:42:30Z")
Have you solved this? I meet the same question:
> **[Google Colaboratory](https://colab.research.google.com/drive/1RKXQhdoP2b3VFa0IQ0wLfil54eb7EppF?usp=sharing)**
---
**Author:**  [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C
**Post date:** [July 26, 2023, 9:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-and-accelerate%2F26382%2F9 "2023-07-26T09:46:48Z")
> [@didi-ai](#):
>
> you
After installing you need to restart the runtime so it loads the newer version
---
**Author:**  [@didi-ai](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdidi-ai)%5C
**Post date:** [July 26, 2023, 9:59am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-and-accelerate%2F26382%2F10 "2023-07-26T09:59:08Z")
Thanks, I changed the runtime type to GPU and do not change any code, then the code is works.
The code still didn’t work on CPU.
Do you know why?
---
**Author:**  [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C
**Post date:** [July 26, 2023, 10:47am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-and-accelerate%2F26382%2F11 "2023-07-26T10:47:53Z")
> [@didi-ai](#):
>
> Do you know why?
On the CPU version, do `pip install accelerate -U`, make sure it shows the latest, and do Runtime → Restart Runtime, then run the code again (skipping the install). Let me know if this still errors out
---
**Author:**  [@didi-ai](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdidi-ai)%5C
**Post date:** [July 27, 2023, 10:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-and-accelerate%2F26382%2F12 "2023-07-27T10:25:31Z")
🤗 Thanks, It works on CPU as you say.
---
**Author:**  [@brando](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbrando)%5C
**Post date:** [September 27, 2023, 9:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-and-accelerate%2F26382%2F13 "2023-09-27T21:48:00Z")
@muellerzr if I want to use only fsdp, do I need HF accelerate? how would I run my script?
---
**Author:**  [@pranil51](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpranil51)%5C
**Post date:** [September 19, 2024, 12:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-and-accelerate%2F26382%2F14 "2024-09-19T12:02:29Z")
I have set accelerate configuration using ‘accelerate config’ in bash. Now does Trainer create some other accelerate configuration by its own?