# Basics for Multi GPU Training with Huggingface Trainer

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbasics-for-multi-gpu-training-with-huggingface-trainer%2F43182%3E%5C
**Category:** 🤗Transformers\
**Created:** [June 14, 2023, 8:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbasics-for-multi-gpu-training-with-huggingface-trainer%2F43182 "2023-06-14T08:46:06Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Scorix](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/scorix/32/14917_2.png) [@Scorix](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FScorix)%5C
**Post date:** [June 14, 2023, 8:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbasics-for-multi-gpu-training-with-huggingface-trainer%2F43182%2F1 "2023-06-14T08:46:06Z")

</div>

After reading the documentation about the trainer [https://huggingface.co/docs/transformers/main\_classes/trainer#pytorch-fully-sharded-data-parallel](https://huggingface.co/docs/transformers/main_classes/trainer#pytorch-fully-sharded-data-parallel)

and further on the documentation about DeepSpeed [https://huggingface.co/docs/transformers/main/main\_classes/deepspeed](https://huggingface.co/docs/transformers/main/main_classes/deepspeed)

I wonder, how the actualy **Python** implementation looks like. Both documentations go in detail about how to setup the SLURM batch, run the `torch.distributed.run` batch script, but I couldn’t find any documentation on how my actual `train.py` should look like.

Given this example script, what do I need to modify, to actually use it for ZeRO MultiGPU (and MultiNode) training? (Using DeepSpeed Integration with the Trainer Class, and ZeRO Stage 1)

```python
model = AutoModelFromCasualLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")
train_data = TextDataset(tokenizer=tokenizer, file_path="data.txt", block_size=128)
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)

training_args = TrainingArguments(
    output_dir = "output",
    overwrite_output_dir=True,
    num_train_epochs=3,
    per_device_train_batch_size=16,
    save_steps=1000,
    save_total_limit=2
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator
)

trainer.train()

trainer.save_model("models/trained_model.pt")

```