# Config.json is not saving after finetuning Llama 2

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconfig-json-is-not-saving-after-finetuning-llama-2%2F56871%3E%5C
**Category:** Beginners\
**Created:** [September 30, 2023, 12:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconfig-json-is-not-saving-after-finetuning-llama-2%2F56871 "2023-09-30T00:38:58Z")\
**Posts on this page:** 1\
**Showing post:** 6

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [May 1, 2024, 2:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconfig-json-is-not-saving-after-finetuning-llama-2%2F56871%2F6 "2024-05-01T14:26:17Z")

</div>

If you train a model with LoRa (low-rank adaptation), you only train adapters on top of the base model. E.g. if you fine-tune LLaMa with LoRa, you only add a couple of linear layers (so-called adapters) on top of the original (also called base) model. Hence calling `save_pretrained()` or `push_to_hub()` will only save 2 things:

- the adapter configuration (in an `adapter_config.json` file)
- the adapter weights (typically in a safetensors file).

See here for example: [ybelkada/opt-350m-lora at main](https://huggingface.co/ybelkada/opt-350m-lora/tree/main). Here, OPT-350m is the base model.

In order to merge these adapter layers back into the base model, one can call the [merge\_and\_unload](https://huggingface.co/docs/peft/v0.10.0/en/package_reference/lora#peft.LoraModel.merge_and_unload) method. Afterwards, you can call `save_pretrained()` on it which will save both the weights and the configuration in a config.json file:

```python
from transformers import AutoModelForCausalLM
from peft import PeftModel

model = AutoModelForCausalLM.from_pretrained(base_model_name)
model = PeftModel.from_pretrained(model, adapter_model_name)

model = model.merge_and_unload()
model.save_pretrained("my_model")

```

One feature of the Transformers library is that it has PEFT integration, which means that you can call `from_pretrained()` directly on a folder/repository that only contains this `adapter_config.json` file and the adapter weights, and it will automatically load the weights of the base model + adapters. See [PEFT integrations](https://huggingface.co/docs/peft/tutorial/peft_integrations#transformers). Hence we could also just have done this:

```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(folder_containing_only_adapter_weights)
model.save_pretrained("my_model")

```

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconfig-json-is-not-saving-after-finetuning-llama-2%2F56871)._