# Where is SageMaker Distributed configured in HF Trainer?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhere-is-sagemaker-distributed-configured-in-hf-trainer%2F6026%3E%5C
**Category:** Amazon SageMaker\
**Created:** [May 6, 2021, 10:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhere-is-sagemaker-distributed-configured-in-hf-trainer%2F6026 "2021-05-06T10:34:31Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![OlivierCR](https://avatars.discourse-cdn.com/v4/letter/o/b19c9b/32.png) [@OlivierCR](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FOlivierCR)%5C
**Post date:** [May 6, 2021, 10:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhere-is-sagemaker-distributed-configured-in-hf-trainer%2F6026%2F1 "2021-05-06T10:34:31Z")

</div>

I see that the HF Trainer [run\_qa script](https://github.com/huggingface/notebooks/blob/master/sagemaker/03_distributed_training_data_parallelism/sagemaker-notebook.ipynb) is compatible with SageMaker Distributed Data Parallel, but I don’t see where is it configured?

In particular, I can see in the [training\_args](https://github.com/huggingface/transformers/blob/f594090a936a06524d2df77c0eb65a42bf694d6b/src/transformers/training_args.py#L42) that smdist gets imported and configured, but where is the model wrapped with smdist DDP?

According to the [smdist doc](https://docs.aws.amazon.com/sagemaker/latest/dg/data-parallel-modify-sdp.html) the below snippet is a required step ; I’d like to understand where it’s done with HF Trainer

```auto
from smdistributed.dataparallel.torch.parallel.distributed import DistributedDataParallel as DDP
model = DDP(Net().to(device))

```

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fphilschmid)%5C
**Post date:** [May 6, 2021, 11:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhere-is-sagemaker-distributed-configured-in-hf-trainer%2F6026%2F2 "2021-05-06T11:34:57Z")

</div>

Hey @OlivierCR,

both the SageMaker Distributed Data-Parallel and the Model-Parallel library are directly integrated into the `Trainer` API, which uses and initializes both libraries automatically.  
For `SMD`:

1. The library is first imported with an alias for the default PyTorch DDP library [here](https://github.com/huggingface/transformers/blob/f594090a936a06524d2df77c0eb65a42bf694d6b/src/transformers/trainer.py#L167)
2. and then wrapps the model [here](https://github.com/huggingface/transformers/blob/f594090a936a06524d2df77c0eb65a42bf694d6b/src/transformers/trainer.py#L957)

P.S. The [\_wrap\_model()](https://github.com/huggingface/transformers/blob/f594090a936a06524d2df77c0eb65a42bf694d6b/src/transformers/trainer.py#L910) function also handles `SMP`

---

<div class="post-metadata">

**Author:** ![OlivierCR](https://avatars.discourse-cdn.com/v4/letter/o/b19c9b/32.png) [@OlivierCR](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FOlivierCR)%5C
**Post date:** [May 6, 2021, 11:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhere-is-sagemaker-distributed-configured-in-hf-trainer%2F6026%2F3 "2021-05-06T11:50:26Z")

</div>

wow so amazing, good job guys