# Config.json is not saving after finetuning Llama 2

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconfig-json-is-not-saving-after-finetuning-llama-2%2F56871%3E%5C
**Category:** Beginners\
**Created:** [September 30, 2023, 12:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconfig-json-is-not-saving-after-finetuning-llama-2%2F56871 "2023-09-30T00:38:58Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![madsthaks](https://avatars.discourse-cdn.com/v4/letter/m/a9adbd/32.png) [@madsthaks](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmadsthaks)%5C
**Post date:** [September 30, 2023, 12:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconfig-json-is-not-saving-after-finetuning-llama-2%2F56871%2F1 "2023-09-30T00:38:58Z")

</div>

After finetuning, i’m not able to save a config.json file using `trainer.model.save_pretrained`

My pip install:

```auto
!pip install torch datasets
!pip install -q accelerate==0.21.0 peft==0.4.0 bitsandbytes==0.40.2 transformers==4.31.0 trl==0.4.7

```

Model code:

```auto
model_name='meta-llama/Llama-2-7b-chat-hf'

model_config = transformers.AutoConfig.from_pretrained(
    model_name,
)

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type='nf4',
    bnb_4bit_compute_dtype=getattr(torch,"float16"),
    bnb_4bit_use_double_quant=False,
)

model = AutoModelForCausalLM.from_pretrained(
   model_name,
    quantization_config=bnb_config,
)

huggingface_dataset_name = "mlabonne/guanaco-llama2-1k"

#dataset = load_dataset(huggingface_dataset_name, "pqa_labeled", split = "train")

dataset = load_dataset(huggingface_dataset_name, split="train")

# LoRA attention dimension
lora_r = 64

# Alpha parameter for LoRA scaling
lora_alpha = 16

# Dropout probability for LoRA layers
lora_dropout = 0.1

# Load LoRA configuration
peft_config = LoraConfig(
    lora_alpha=lora_alpha,
    lora_dropout=lora_dropout,
    r=lora_r,
    bias="none",
    task_type="CAUSAL_LM",
)

training_arguments = TrainingArguments(
    output_dir="random_weights",
    fp16=True,
    learning_rate=1e-5,
    num_train_epochs=5,
    weight_decay=0.01,
    logging_steps=1,
    max_steps=1)

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    peft_config=peft_config,
    dataset_text_field="text",
    max_seq_length=None,
    tokenizer=tokenizer,
    args=training_arguments
)

# Train model
trainer.train()

trainer.model.save_pretrained("finetuned_llama")

```

Now, when I go to load the model it complaing about not having a config.json. I’ve also tried explicitly saving the config using `trainer.model.config.save_pretrained` but I had no luck.

I might be missing something obvious. Any ideas on what might be going wrong?

---

<div class="post-metadata">

**Author:** ![pabligme](https://avatars.discourse-cdn.com/v4/letter/p/eb9ed0/32.png) [@pabligme](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpabligme)%5C
**Post date:** [November 14, 2023, 3:40pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconfig-json-is-not-saving-after-finetuning-llama-2%2F56871%2F2 "2023-11-14T15:40:11Z")

</div>

Did you fix that?

---

<div class="post-metadata">

**Author:** ![hfreedma](https://avatars.discourse-cdn.com/v4/letter/h/59ef9b/32.png) [@hfreedma](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhfreedma)%5C
**Post date:** [December 4, 2023, 5:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconfig-json-is-not-saving-after-finetuning-llama-2%2F56871%2F3 "2023-12-04T17:44:26Z")

</div>

I think it is because you are using LoRA which trains an adapter model. Try calling `model.merge_and_unload()` to merge the adapter model with your base model before saving.

---

<div class="post-metadata">

**Author:** ![hemanthkumar23](https://avatars.discourse-cdn.com/v4/letter/h/6de8d8/32.png) [@hemanthkumar23](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhemanthkumar23)%5C
**Post date:** [May 1, 2024, 2:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconfig-json-is-not-saving-after-finetuning-llama-2%2F56871%2F4 "2024-05-01T14:15:04Z")

</div>

Same problem with me also.

---

<div class="post-metadata">

**Author:** ![hemanthkumar23](https://avatars.discourse-cdn.com/v4/letter/h/6de8d8/32.png) [@hemanthkumar23](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhemanthkumar23)%5C
**Post date:** [May 1, 2024, 2:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconfig-json-is-not-saving-after-finetuning-llama-2%2F56871%2F5 "2024-05-01T14:15:41Z")

</div>

Can you please elaborate?

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [May 1, 2024, 2:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconfig-json-is-not-saving-after-finetuning-llama-2%2F56871%2F6 "2024-05-01T14:26:17Z")

</div>

If you train a model with LoRa (low-rank adaptation), you only train adapters on top of the base model. E.g. if you fine-tune LLaMa with LoRa, you only add a couple of linear layers (so-called adapters) on top of the original (also called base) model. Hence calling `save_pretrained()` or `push_to_hub()` will only save 2 things:

- the adapter configuration (in an `adapter_config.json` file)
- the adapter weights (typically in a safetensors file).

See here for example: [ybelkada/opt-350m-lora at main](https://huggingface.co/ybelkada/opt-350m-lora/tree/main). Here, OPT-350m is the base model.

In order to merge these adapter layers back into the base model, one can call the [merge\_and\_unload](https://huggingface.co/docs/peft/v0.10.0/en/package_reference/lora#peft.LoraModel.merge_and_unload) method. Afterwards, you can call `save_pretrained()` on it which will save both the weights and the configuration in a config.json file:

```python
from transformers import AutoModelForCausalLM
from peft import PeftModel

model = AutoModelForCausalLM.from_pretrained(base_model_name)
model = PeftModel.from_pretrained(model, adapter_model_name)

model = model.merge_and_unload()
model.save_pretrained("my_model")

```

One feature of the Transformers library is that it has PEFT integration, which means that you can call `from_pretrained()` directly on a folder/repository that only contains this `adapter_config.json` file and the adapter weights, and it will automatically load the weights of the base model + adapters. See [PEFT integrations](https://huggingface.co/docs/peft/tutorial/peft_integrations#transformers). Hence we could also just have done this:

```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(folder_containing_only_adapter_weights)
model.save_pretrained("my_model")

```

---

<div class="post-metadata">

**Author:** ![m-faraz-ali](https://avatars.discourse-cdn.com/v4/letter/m/d78d45/32.png) [@m-faraz-ali](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fm-faraz-ali)%5C
**Post date:** [May 9, 2024, 12:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconfig-json-is-not-saving-after-finetuning-llama-2%2F56871%2F7 "2024-05-09T12:59:57Z")

</div>

I got the following error while running the upper code,

PS C:\Users\Anonymous\Desktop\AIRL Lab\MedLLM\> python -u “c:\Users\Anonymous\Desktop\AIRL Lab\MedLLM\HF Connection (Temporary Use)\merging.py”  
config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 699/699 [00:00\<?, ?B/s]  
model.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████| 2.20G/2.20G [14:18\<00:00, 2.56MB/s]  
generation\_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████| 124/124 [00:00\<?, ?B/s]  
Traceback (most recent call last):  
File “c:\Users\Anonymous\Desktop\AIRL Lab\MedLLM\HF Connection (Temporary Use)\merging.py”, line 8, in   
model = PeftModel.from\_pretrained(model, adapter\_model\_name)  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “C:\Users\Anonymous\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11\_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\peft\peft\_model.py”, line 356, in from\_pretrained  
model.load\_adapter(model\_id, adapter\_name, is\_trainable=is\_trainable, \*\*kwargs)  
File “C:\Users\Anonymous\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11\_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\peft\peft\_model.py”, line 727, in load\_adapter  
adapters\_weights = load\_peft\_weights(model\_id, device=torch\_device, \*\*hf\_hub\_download\_kwargs)  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “C:\Users\Anonymous\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11\_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\peft\utils\save\_and\_load.py”, line 326, in load\_peft\_weights  
adapters\_weights = safe\_load\_file(filename, device=device)  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “C:\Users\Anonymous\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11\_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\safetensors\torch.py”, line 311, in load\_file  
with safe\_open(filename, framework=“pt”, device=device) as f:  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
safetensors\_rust.SafetensorError: Error while deserializing header: InvalidHeaderDeserialization  
PS C:\Users\Anonymous\Desktop\AIRL Lab\MedLLM\>

---

<div class="post-metadata">

**Author:** ![Owos](https://avatars.discourse-cdn.com/v4/letter/o/87869e/32.png) [@Owos](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FOwos)%5C
**Post date:** [October 11, 2024, 7:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconfig-json-is-not-saving-after-finetuning-llama-2%2F56871%2F8 "2024-10-11T19:34:23Z")

</div>

> [@nielsr](#):
>
> ```auto
> om transformers import AutoModelForCausalLM
> 
> model = AutoModelForCausalLM.from_pretrained(folder_containing_only_adapter_weights)
> model = model.merge_and_unload()
> 
> ```

I tried this part :

```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(folder_containing_only_adapter_weights)
model = model.merge_and_unload()
model.save_pretrained("my_model")

```

and I got this error .  
`AttributeError: 'LlamaForCausalLM' object has no attribute 'merge_and_unload'`

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [October 12, 2024, 11:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconfig-json-is-not-saving-after-finetuning-llama-2%2F56871%2F9 "2024-10-12T11:01:14Z")

</div>

Hi,

It looks like one can only call the merge\_and\_unload() method on the PeftModel class available in the PEFT library, not on a class in Transformers. So the only way to merge adapters into the base model is using this:

```python
model = AutoModelForCausalLM.from_pretrained(base_model_name)
model = PeftModel.from_pretrained(model, adapter_model_name)

model = model.merge_and_unload()
model.save_pretrained("merged_adapters")

```

---

<div class="post-metadata">

**Author:** ![Owos](https://avatars.discourse-cdn.com/v4/letter/o/87869e/32.png) [@Owos](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FOwos)%5C
**Post date:** [October 22, 2024, 2:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconfig-json-is-not-saving-after-finetuning-llama-2%2F56871%2F10 "2024-10-22T02:34:19Z")

</div>

okay then, thank you!