# Accelerator.backward(loss) never done!
**URL:**
**Author:**  [@Meyta](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMeyta)%5C
**Post date:** [March 9, 2023, 2:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerator-backward-loss-never-done%2F33387%2F1 "2023-03-09T02:54:18Z")
I am new to accelerate, my code got stuck in accelerator.backward(loss) forever . I used a sample code for finetuning a language model as in the bottom of my question.
Also, the accelerator config is set to have 4 gpus, but it return 1 when I print in the code `print(accelerator.num_processes)`.
Anyone has any idea about my questions?
Thank you all 🙂
```auto
from accelerate import Accelerator
import torch
torch.cuda.empty_cache()
from transformers import AutoTokenizer
from datasets import load_dataset
from torch.optim import AdamW
from transformers import get_scheduler
from transformers import AutoModelForSequenceClassification
from torch.utils.data import DataLoader
accelerator = Accelerator()
dataset = load_dataset("yelp_review_full")
dataset["train"][100]
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
def tokenize_function(examples):
return tokenizer(examples["text"], padding="max_length", truncation=True)
tokenized_datasets = dataset.map(tokenize_function, batched=True)
tokenized_datasets = tokenized_datasets.remove_columns(["text"])
tokenized_datasets = tokenized_datasets.rename_column("label", "labels")
tokenized_datasets.set_format("torch")
small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(100))
small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(100))
train_dataloader = DataLoader(small_train_dataset, shuffle=True, batch_size=24)
eval_dataloader = DataLoader(small_eval_dataset, batch_size=8)
model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=5)
optimizer = AdamW(model.parameters(), lr=5e-5)
num_epochs = 1
num_training_steps = num_epochs * len(train_dataloader)
lr_scheduler = get_scheduler(
name="linear", optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps
)
model, optimizer, train_dataloader, eval_dataloader, lr_scheduler = accelerator.prepare(model, optimizer, train_dataloader, eval_dataloader, lr_scheduler)
print(model.device)
from tqdm.auto import tqdm
progress_bar = tqdm(range(num_training_steps))
model.train()
for epoch in range(num_epochs):
for batch in train_dataloader:
outputs = model(**batch)
print(outputs)
loss = outputs.loss
print(loss)
accelerator.backward(loss)
# loss.backward()
optimizer.step()
lr_scheduler.step()
optimizer.zero_grad()
progress_bar.update(1)
```
---
**Author:**  [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C
**Post date:** [March 9, 2023, 3:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerator-backward-loss-never-done%2F33387%2F2 "2023-03-09T03:24:15Z")
> [@Meyta](#):
>
> Also, the accelerator config is set to have 4 gpus, but it return 1 when I print in the code
Ill need the output of `accelerate env` and the command you’re using to run the code as well, and ideally your version of the library
---
**Author:**  [@Meyta](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMeyta)%5C
**Post date:** [March 9, 2023, 3:47am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerator-backward-loss-never-done%2F33387%2F3 "2023-03-09T03:47:37Z")
I use `accelerate launch sample_bert.py` to run and Here is the output of accelerate env:
```auto
- `Accelerate` version: 0.16.0
- Platform: Linux-5.15.0-1027-gcp-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- Numpy version: 1.21.6
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- `Accelerate` default config:
- compute_environment: LOCAL_MACHINE
- distributed_type: MULTI_GPU
- mixed_precision: fp16
- use_cpu: False
- dynamo_backend: NO
- num_processes: 4
- machine_rank: 0
- num_machines: 1
- gpu_ids: 0
- rdzv_backend: static
- same_network: True
- main_training_function: main
- deepspeed_config: {}
- fsdp_config: {}
- megatron_lm_config: {}
- downcast_bf16: no
```
---
**Author:**  [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C
**Post date:** [March 9, 2023, 4:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerator-backward-loss-never-done%2F33387%2F4 "2023-03-09T04:06:25Z")
> [@Meyta](#):
>
> `gpu_ids: 0`
This needs to be “all” (the default). Youre telling it to just use the first GPU