# LoRA training with accelerate / deepspeed
**URL:**
**Author:**  [@iarbel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fiarbel)%5C
**Post date:** [January 22, 2024, 11:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flora-training-with-accelerate-deepspeed%2F70080%2F1 "2024-01-22T11:54:50Z")
I’m training LoRA adapters large models using Accelerate + DeepSpeed, with ZeRO-3. I’m using alignment-handbook implementation for that. However, when saving checkpoints, the full model is being saved, while I need only the adapter (and possibly the optimizer states). Is this possible to configure the training such that save only the adapters are saved?
Or should I used a MULTI\_GPU setup instead of DeepSpeed?
Model to be trained is Llama-70B, on 8\*A100 40GB, so it’s really not possible to fit the model on a single GPU.
---
**Author:**  [@blazejm](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fblazejm)%5C
**Post date:** [August 8, 2024, 7:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flora-training-with-accelerate-deepspeed%2F70080%2F2 "2024-08-08T07:50:27Z")
hey, did you figure it out?
---
**Author:**  [@iarbel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fiarbel)%5C
**Post date:** [August 8, 2024, 9:20am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flora-training-with-accelerate-deepspeed%2F70080%2F3 "2024-08-08T09:20:02Z")
Use multi-GPU and not DeepSpeed
---
**Author:**  [@MauroExtrac](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMauroExtrac)%5C
**Post date:** [May 28, 2025, 12:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flora-training-with-accelerate-deepspeed%2F70080%2F4 "2025-05-28T12:30:56Z")
@iarbel What do you mean by using multi GPU rather than DeepSpeed?