# ZeRO uses more RAM than DDP?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fzero-uses-more-ram-than-ddp%2F49830%3E%5C
**Category:** DeepSpeed\
**Created:** [August 7, 2023, 2:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fzero-uses-more-ram-than-ddp%2F49830 "2023-08-07T14:46:56Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kenkentron](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@Kenkentron](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKenkentron)%5C
**Post date:** [August 7, 2023, 2:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fzero-uses-more-ram-than-ddp%2F49830%2F1 "2023-08-07T14:46:56Z")

</div>

Hi community,

I am messing around with this example ([DeepSpeed](https://huggingface.co/docs/peft/accelerate/deepspeed-zero3-offload)) and the full code is here ([https://github.com/huggingface/peft/blob/main/examples/conditional\_generation/peft\_lora\_seq2seq\_accelerate\_ds\_zero3\_offload.py](https://github.com/huggingface/peft/blob/main/examples/conditional_generation/peft_lora_seq2seq_accelerate_ds_zero3_offload.py)) and have a question regarding memory usage.

Specifically, I was interested in how much ZeRO3 was able to reduce memory usage compared to running the same training script 1) on 1 GPU vs 2) using Pytorch DDP (example: [From PyTorch DDP to Accelerate to Trainer, mastery of distributed training with ease](https://huggingface.co/blog/pytorch-ddp-accelerate-transformers)).

I am using 2 \* 2080ti and observe the following,

1. If I run `python peft_lora_seq2seq_accelerate_ds_zero3_offload.py`, my first GPU roughly uses 2500-2900MB RAM;
2. If I run `torchrun --nproc-per-node=2 peft_lora_seq2seq_accelerate_ds_zero3_offload.py`, both of my GPUs use 2500-2900MB RAM; which makes sense as DDP just replicates the entire model on both cards;
3. If I run `accelerate launch --config_file ds_zero3_cpu.yaml peft_lora_seq2seq_accelerate_ds_zero3_offload.py` both my GPUs usage are constantly above 3000MB sometimes 5000MB.

I feel like I must have missed something basic here but not quite sure where to look for the problem so any pointers appreciated!

Thanks,