# Which data parallel does trainer use? DP or DDP?
**URL:**
**Author:**  [@xiami](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fxiami)%5C
**Post date:** [March 24, 2022, 6:03am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhich-data-parallel-does-trainer-use-dp-or-ddp%2F16021%2F1 "2022-03-24T06:03:27Z")
I try to search in the doc. But I didn’t find the answer anywhere.
Thank you
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [March 24, 2022, 12:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhich-data-parallel-does-trainer-use-dp-or-ddp%2F16021%2F2 "2022-03-24T12:22:07Z")
It depends if you launch your training script with `python` (in which case it will use DP) or `python -m torch.distributed.launch` (in which case it will use DDP).
---
**Author:**  [@brando](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbrando)%5C
**Post date:** [August 17, 2022, 3:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhich-data-parallel-does-trainer-use-dp-or-ddp%2F16021%2F3 "2022-08-17T15:03:18Z")
perhaps useful to you: [Using Transformers with DistributedDataParallel — any examples?](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-transformers-with-distributeddataparallel-any-examples%2F10775)
---
**Author:**  [@RylanSchaeffer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRylanSchaeffer)%5C
**Post date:** [August 30, 2025, 1:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhich-data-parallel-does-trainer-use-dp-or-ddp%2F16021%2F4 "2025-08-30T01:34:06Z")
I know this is a bit of an old thread, but I have a follow up question. I’m creating a `Trainer()` , evaluating, training and evaluating again. Here’s a snippet of my code:
```
trainer = Trainer(
model=model,
processing\_class=tokenizer,
args=pretraining\_config,
train\_dataset=train\_dataset,
eval\_dataset=eval\_dataset,
data\_collator=data\_collator,
)
logging.info(“Evaluating before training…”)
eval\_metrics\_before = trainer.evaluate()
wandb.log({f"eval\_before/{k}": v for k, v in eval\_metrics\_before.items()})
pprint.pprint(eval\_metrics\_before)
logging.info(“Beginning training…”)
trainer.train()
logging.info(“Finished training. Beginning final evaluation…”)
eval\_metrics\_after = trainer.evaluate()
wandb.log({f"eval\_after/{k}": v for k, v in eval\_metrics\_after.items()})
pprint.pprint(eval\_metrics\_after)
```
When I run with two GPUs and a model small enough to fit on each, I noticed while the job is running that evaluating appears to use data parallelism over the two visible GPUs, but does not for training. Do you know what might cause that or how to fix it?
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [August 30, 2025, 2:42am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhich-data-parallel-does-trainer-use-dp-or-ddp%2F16021%2F5 "2025-08-30T02:42:00Z")
Hmm… Have you tried [launching it via `accelerate` or `torchrun`](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-run-single-node-multi-gpu-training-with-hf-trainer%2F19503)%3F
```bash
# single node, 2 GPUs
torchrun --nproc_per_node=2 train.py
# or
accelerate launch --num_processes=2 train.py
```
### [Accelerator selection](https://huggingface.co/docs/transformers/v4.56.0/en/accelerator_selection)
---
**Author:**  [@RylanSchaeffer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRylanSchaeffer)%5C
**Post date:** [August 30, 2025, 4:23am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhich-data-parallel-does-trainer-use-dp-or-ddp%2F16021%2F6 "2025-08-30T04:23:56Z")
> [@John6666](#):
>
> Hmm… Have you tried [launching it via `accelerate` or `torchrun`](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-run-single-node-multi-gpu-training-with-hf-trainer%2F19503)%3F
Yeah, I would’ve thought that launching with `python` would use DP and thus would only use 1 available GPU. And that’s partially correct: `train()` indeed only uses 1 GPU, but `evaluate()` uses 2 GPUs. Hence my confusion…
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [August 30, 2025, 5:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhich-data-parallel-does-trainer-use-dp-or-ddp%2F16021%2F7 "2025-08-30T05:25:09Z")
I see. When running distributed training, if you [launch it as a single process, `evaluate` sometimes behaves differently from the Trainer part](https://github.com/huggingface/transformers/issues/28956)…Since [`DP` itself seems quite fragile](https://discuss.pytorch.org/t/bug-in-dataparallel-only-works-if-the-dataset-device-is-cuda-0/28634), using `DDP` is probably the simpler approach…