# How to use FSDP + DPP in Trainer

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-fsdp-dpp-in-trainer%2F34064%3E%5C
**Category:** 🤗Transformers\
**Created:** [March 17, 2023, 11:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-fsdp-dpp-in-trainer%2F34064 "2023-03-17T23:16:58Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![maxBing12345](https://avatars.discourse-cdn.com/v4/letter/m/b487fb/32.png) [@maxBing12345](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FmaxBing12345)%5C
**Post date:** [March 17, 2023, 11:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-fsdp-dpp-in-trainer%2F34064%2F1 "2023-03-17T23:16:58Z")

</div>

Hi - I want to train a model with [e.g. 256 GPU]. I want to have 4 data parallelism (DDP) to replicate the full model, and in each parallelism use FSDP to shard the model into 64 GPUs. Any code example?

I know how to write it in a native Pytorch but how to do this in Trainer. Is it supportive?

---

<div class="post-metadata">

**Author:** ![Atharva](https://avatars.discourse-cdn.com/v4/letter/a/b5a626/32.png) [@Atharva](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAtharva)%5C
**Post date:** [April 24, 2023, 11:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-fsdp-dpp-in-trainer%2F34064%2F2 "2023-04-24T23:38:34Z")

</div>

Did you figure it out?