# Proper Adapter Loading for SFT and DPO
**URL:**
**Author:**  [@nyusen](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnyusen)%5C
**Post date:** [January 7, 2025, 9:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fproper-adapter-loading-for-sft-and-dpo%2F134722%2F1 "2025-01-07T21:21:45Z")
I am trying to run multiple rounds of fine tuning on llama2-7b-chat. I completed the first round of fine tuning and got adapter weights (let’s say I named the model `name/llama-2-7b-chat-guanaco`). I’m starting to get confused during the second round of fine tuning, in which I want to fine-tune on top of the already-fine-tuned model. I assumed that I’d be able to load the fine-tuned model like so and run the DPO pipeline:
```auto
model = AutoModelForCausalLM.from_pretrained("name/llama-2-7b-chat-guanaco", quantization_config=quant_config, device_map={"": 0})
model = PeftModel.from_pretrained(model, "name/llama-2-7b-chat-guanaco")
```
I trained and was able to push this new model to huggingface (let’s call it `name/llama-2-7b-chat-guanaco-dpo`). Oddly, the safetensors file is twice as large as the first pass, so I would appreciate some clarity on whether this is expected. But now I’m trying to serve that model for inference. Do I need to load the models in order to correctly sequence the adapters? Something like this:
```auto
first_finetune_path = "name/llama-2-7b-chat-guanaco"
config = PeftConfig.from_pretrained(first_finetune_path)
base_model = AutoModelForCausalLM.from_pretrained(
config.base_model_name_or_path,
load_in_8bit=True,
device_map="auto"
)
self.model = PeftModel.from_pretrained(base_model, first_finetune_path)
# Path to the second fine-tuned model (DPO adapters)
second_finetune_path = "name/llama-2-7b-chat-guanaco-dpo"
# Load the second fine-tuned adapters (DPO)
self.model = PeftModel.from_pretrained(self.model, second_finetune_path)
```
Does it not need to be in sequence? I would appreciate any guidance on how to think about adapters. Thanks!
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [January 9, 2025, 3:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fproper-adapter-loading-for-sft-and-dpo%2F134722%2F2 "2025-01-09T15:09:07Z")
> Oddly, the safetensors file is twice as large as the first pass
Is the model saved in fp32 precision? If so, I think the size will be doubled.
>
>
> \### System Info
>
> \- \`transformers\` version: 4.28.0.dev0
> \- Platform: Linux-4.18….0-372.32.1.el8\_6.x86\_64-x86\_64-with-glibc2.17
> \- Python version: 3.8.16
> \- Huggingface\_hub version: 0.13.3
> \- Safetensors version: not installed
> \- PyTorch version (GPU?): 1.12.1+cu116 (True)
> \- Tensorflow version (GPU?): not installed (NA)
> \- Flax version (CPU?/GPU?/TPU?): not installed (NA)
> \- Jax version: not installed
> \- JaxLib version: not installed
> \- Using GPU in script?: yes
> \- Using distributed or parallel set-up in script?: yes
>
>
> \### Who can help?
>
> @stas00 @sgugger
>
>
>
> \### Information
>
> \- \[X\] The official example scripts
> \- \[X\] My own modified scripts
>
> \### Tasks
>
> \- \[x\] An officially supported task in the \`examples\` folder (such as GLUE/SQuAD, ...)
> \- \[\] My own task or dataset (give details below)
>
> \### Reproduction
>
> I'm using Trainer with deepspeed integration to fine-tune a Llama model.
>
> This is the stage2 config im using:
> \`\`\`json
> {
> "optimizer": {
> "type": "AdamW",
> "params": {
> "lr": "auto",
> "betas": "auto",
> "eps": "auto",
> "weight\_decay": "auto"
> }
> },
>
>
> "zero\_optimization": {
> "stage": 2,
> "offload\_optimizer": {
> "device": "cpu",
> "pin\_memory": true
> },
> "allgather\_partitions": true,
> "allgather\_bucket\_size": 2e8,
> "overlap\_comm": true,
> "reduce\_scatter": true,
> "reduce\_bucket\_size": 2e8,
> "contiguous\_gradients": true
> },
>
> "gradient\_accumulation\_steps": "auto",
> "gradient\_clipping": "auto",
> "train\_batch\_size": "auto",
> "train\_micro\_batch\_size\_per\_gpu": "auto"
> }
> \`\`\`
>
> So I'm using zero2 with optimizer offload. I found the size of the model checkpoints after \`trainer.train()\` become much larger than what they should be.
>
> Using official \[run\_clm.py\](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run\_clm.py) script as an example :
>
> \`\`\`bash
> deepspeed --num\_gpus=1 run\_clm.py \\
> --num\_train\_epochs 0.01 \\
> --model\_name\_or\_path decapoda-research/llama-7b-hf \\
> --dataset\_name wikitext \\
> --dataset\_config\_name wikitext-2-raw-v1 \\
> --per\_device\_train\_batch\_size 2 \\
> --do\_train \\
> --output\_dir /tmp/test-plm \\
> --deepspeed ds\_config.json
> \`\`\`
>
> I add these two save\_model lines around \`trainer.train()\` for testing:
> \`\`\`python
> trainer.save\_model("test1")
> train\_result = trainer.train(resume\_from\_checkpoint=checkpoint)
> trainer.save\_model("test2")
> \`\`\`
>
> Now check the size:
> \`\`\`bash
> du -sh test1
> 26G test1
>
> du -sh test2
> 76G test2
> \`\`\`
> Note, I have deleted \`global\_step\*\` folder in \`test2\` before calculating the size.
>
> I believe 26G is the correct size for an fp32 llama 7b. So, after training with trainer, the model size is wrong? Interestingly, seems the wrong size model still works with \`.from\_pretrain\`.
>
>
> I have located the issue raised after this \[line\](https://github.com/huggingface/transformers/blob/dacd34568d1a27b91f84610eab526640ed8f94e0/src/transformers/deepspeed.py#L378), which changed the model assignment in trainer \`\_inner\_training\_loop\` \[here\](https://github.com/huggingface/transformers/blob/dacd34568d1a27b91f84610eab526640ed8f94e0/src/transformers/trainer.py#L1733) afterward. After this the model saved by \`trainer.\_save()\` will have the wrong size.
>
> Does deepspeed engine add some extra things to pytorch\_model.bin? is this expected?
>
> My current solution to this is always using \`self.deepspeed.save\_16bit\_model()\` in \[trainer.save\_model()\](https://github.com/huggingface/transformers/blob/dacd34568d1a27b91f84610eab526640ed8f94e0/src/transformers/trainer.py#L2771) for zerostage2:
> \`\`\`python
> elif self.deepspeed:
> # this takes care of everything as long as we aren't under zero3
> if self.args.should\_save:
> self.\_save(output\_dir)
> if is\_deepspeed\_zero3\_enabled():
> # It's too complicated to try to override different places where the weights dump gets
> # saved, so since under zero3 the file is bogus, simply delete it. The user should
> # either user deepspeed checkpoint to resume or to recover full weights use
> # zero\_to\_fp32.py stored in the checkpoint.
> if self.args.should\_save:
> file = os.path.join(output\_dir, WEIGHTS\_NAME)
> if os.path.isfile(file):
> # logger.info(f"deepspeed zero3: removing {file}, see zero\_to\_fp32.py to recover weights")
> os.remove(file)
>
> # now save the real model if stage3\_gather\_16bit\_weights\_on\_model\_save=True
> # if false it will not be saved.
> # This must be called on all ranks
> if not self.deepspeed.save\_16bit\_model(output\_dir, WEIGHTS\_NAME):
> logger.warning(
> "deepspeed.save\_16bit\_model didn't save the model, since"
> " stage3\_gather\_16bit\_weights\_on\_model\_save=false. Saving the full checkpoint instead, use"
> " zero\_to\_fp32.py to recover weights"
> )
> self.deepspeed.save\_checkpoint(output\_dir)
> else:
> if self.args.should\_save:
> for filename in os.listdir(output\_dir):
> full\_filename = os.path.join(output\_dir, filename)
> # If we have a shard file that is not going to be replaced, we delete it, but only from the main process
> # in distributed settings to avoid race conditions.
> weights\_no\_suffix = WEIGHTS\_NAME.replace(".bin", "").replace(".safetensors", "")
>
> # delete everything start with weights\_no\_suffix, usually are "pytorch\_model".
> if (
> filename.startswith(weights\_no\_suffix)
> and os.path.isfile(full\_filename)
> ):
> os.remove(full\_filename)
> self.deepspeed.save\_16bit\_model(output\_dir, WEIGHTS\_NAME)
> \`\`\`
>
>
>
>
> \### Expected behavior
>
> Model checkpoint size should be unchanged after \`trainer.train()\`
> Does it not need to be in sequence? I would appreciate any guidance on how to think about adapters.
While I can’t say for sure that it’s completely unrelated, in general, changing the order in which you apply the adapters rarely causes any problems.
Whether you call it an adapter or LoRA, PEFT is in charge of that part in Hugging Face, so you can understand the whole story by reading PEFT-related literature.
> **[LoRA](https://huggingface.co/docs/peft/main/en/developer_guides/lora)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
By the way, there are cases where errors occur when applying LoRA in the model quantized state, so if you have any problems applying LoRA in the model quantized state, try applying it after first de-quantizing it, and then quantizing it again.