# Further finetuning a LoRA finetuned CausalLM Model
**URL:**
**Author:**  [@JulianGerhard](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJulianGerhard)%5C
**Post date:** [April 18, 2023, 12:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-finetuning-a-lora-finetuned-causallm-model%2F36987%2F1 "2023-04-18T12:44:22Z")
Hey everyone,
I am a bit unsure how to proceed regarding the mentioned topic.
The baseline is a model created via Huggingface’s library as an AutoModelForCausalLM model, PEFT and a LoRA approach with subsequent merging of the weights.
I now want to further fine tune the model without losing its original properties - in this case via instruction fine tuning or prefix tuning.
My approach would be the following:
```auto
model = AutoModelForCausalLM.from_pretrained(
model_id,
use_cache=False if gradient_checkpointing else True
device_map="auto",
load_in_8bit=True,
)
model = create_peft_config(model)
output_dir = "/tmp"
training_args = TrainingArguments(
output_dir=output_dir,
overwrite_output_dir=True,
per_device_train_batch_size=per_device_train_batch_size,
per_device_eval_batch_size=per_device_train_batch_size,
bf16=bf16,
learning_rate=lr,
num_train_epochs=epochs,
gradient_checkpointing=gradient_checkpointing,
gradient_accumulation_steps=2,
logging_dir=f"{output_dir}/logs",
logging_strategy="steps",
logging_steps=10,
optim="adafactor",
save_strategy="epoch",
save_total_limit=3,
evaluation_strategy="epoch",
load_best_model_at_end=False,
no_cuda=False,
auto_find_batch_size=True
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=dataset_train,
compute_metrics=compute_metrics,
preprocess_logits_for_metrics=preprocess_logits_for_metrics,
eval_dataset=dataset_eval,
data_collator=default_data_collator
)
trainer.train()
trainer.model.save_pretrained(output_dir)
del model
del trainer
peft_config = PeftConfig.from_pretrained(output_dir)
model = AutoModelForCausalLM.from_pretrained(
peft_config.base_model_name_or_path,
load_in_8bit=False,
return_dict=True,
device_map="auto",
torch_dtype=torch.float16,
low_cpu_mem_usage=True,
)
model = PeftModel.from_pretrained(
model,
output_dir,
torch_dtype=torch.float16,
device_map="auto",
)
model.eval()
os.makedirs("lora", exist_ok=True)
merged_model = model.merge_and_unload()
merged_model.save_pretrained('lora')
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.save_pretrained('lora')
```
In principle, I am loading the original model with the merged weights, finetune that on new data likewise with PEFT and LoRA and afterwards merging the weights again into the base model.
Is this a sensible approach, or is there something to suggest, for example, that I might even significantly compromise the original capabilities by doing so? If something speaks against it, what would be a better approach?
Kind regards and thanks in advance
---
**Author:**  [@NajiAboo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNajiAboo)%5C
**Post date:** [July 14, 2023, 9:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-finetuning-a-lora-finetuned-causallm-model%2F36987%2F2 "2023-07-14T09:56:43Z")
Did you able to figure out it? I am also looking for the same
---
**Author:**  [@capnchat](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcapnchat)%5C
**Post date:** [December 27, 2023, 2:36am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-finetuning-a-lora-finetuned-causallm-model%2F36987%2F3 "2023-12-27T02:36:08Z")
I am also interested in fine-tuning a CasualLM model with Peft, saving it, then continue fine-tuning on a different dataset, and repeating this pattern in order to avoid the scenario where the training fails after an extended period, which would waste the experiment, as in my case it will take a long time to train on the entire dataset. I am getting errors when attempting to further fine-tune the previously-fine-tuned model, but now am also seeing posts stating that it is advisable to train the model on a combined dataset instead all at once to ensure the greater patterns are learned.
So even if I could figure out how to iteratively fine-tune, is it even advisable?
Now I am reading about merging multiple fine-tuned models, but will this also risk affecting the performance and should we just simply train the model on the entire dataset all at once?
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [December 27, 2023, 11:23am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-finetuning-a-lora-finetuned-causallm-model%2F36987%2F4 "2023-12-27T11:23:26Z")
Hi,
I think there are several possibilities:
- adding an adapter which you train on task 1, then merge with the base model, and add another adapter which is trained on task 2.
- adding an adapter which you train on one dataset, then you train it on another dataset, etc. (so only 1 adapter). Eventually you could merge it into the base model.
- adding and training several adapters (one for each task/dataset) separately and merge them simultanously
For the first case, if you fine-tuned an `xxxForCausalLM` model using PEFT (i.e. by adding an adapter), you can load the model with its adapter using the `AutoPeftModelForCausalLM` class. Note that the adapter weights will still be separated from the base model. You could merge the adapter weights into the base model by calling the [merge\_and\_unload](https://huggingface.co/docs/peft/v0.7.1/en/package_reference/lora#peft.LoraModel.merge_and_unload) method. Next, you could add another adapter, and apply PEFT again.
Let’s show this in code.
## Step 1: load base model + adapter weights
First of all, note that there are 2 ways to load a base model with its adapter weights:
```auto
from peft import PeftModel, PeftConfig, AutoPeftModelForCausalLM
from transformers import AutoModelForCausalLM
# let's say you fine-tuned OPT using PEFT
# method 1: separately
base_model_id = "facebook/opt-350m"
adapter_id = "ybelkada/opt-350m-lora"
base_model = AutoModelForCausalLM.from_pretrained(base_model_id)
base_with_adapters_model = PeftModel.from_pretrained(base_model, adapter_id)
# method 2: conveniently with the AutoPeftModelForCausalLM class
base_with_adapters_model = AutoPeftModelForCausalLM.from_pretrained("ybelkada/opt-350m-lora")
```
Note that in both cases, the adapter weights are still stored separately from the base model (you can see this in the state dictionary, which still includes separate `base_model` and `adapter` keys).
## Step 2: merge adapter weights into the base model
Hence, you can merge the adapter parameters with the base model:
```auto
# now we just have a regular AutoModelForCausalLM Transformers model
model = base_with_adapters_model.merge_and_unload()
```
## Step 3: add another adapter
Next, we could apply PEFT again by adding another adapter:
```auto
# next, we could apply PEFT again by adding another adapter
from peft import get_peft_model, LoraConfig, TaskType
lora_config = LoraConfig(
r=16,
target_modules=["q_proj", "v_proj"],
task_type=TaskType.CAUSAL_LM,
lora_alpha=32,
lora_dropout=0.05
)
base_model_with_new_adapter = get_peft_model(model, lora_config)
base_model_with_new_adapter.print_trainable_parameters()
```
You can fine-tune the `base_model_with_new_adapter` using the Trainer API or PyTorch.
See this guide for more info: [PEFT integrations](https://huggingface.co/docs/peft/tutorial/peft_integrations).
---
**Author:**  [@capnchat](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcapnchat)%5C
**Post date:** [December 27, 2023, 4:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-finetuning-a-lora-finetuned-causallm-model%2F36987%2F5 "2023-12-27T16:59:51Z")
Thanks a lot for your reply @nielsr.
I was not aware that a model trained with PEFT is considered an “adapter”, and thought it was generally the same as a regular fine-tuned model, as I’ve been able to load the saved pretrained model that was trained with PEFT using AutoModelForCausalLM and passing the local path where the PEFT model was saved, and generate inferences with it, as opposed to loading it using PeftModel.from\_pretrained() and passing both the adapter ID as well as a loaded base model.
I see the second method you offered only requires the adapter ID, so anyways this gives me a lot to consider and look into and I really appreciate your reply.
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [December 27, 2023, 6:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-finetuning-a-lora-finetuned-causallm-model%2F36987%2F6 "2023-12-27T18:33:12Z")
Hi,
Well PEFT is actually integrated into the Transformers library, see here: [PEFT integrations](https://huggingface.co/docs/peft/tutorial/peft_integrations#transformers).
This means that if you do the following:
```auto
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("ybelkada/opt-350m-lora")
```
it will automatically load the base model + adapter weights (as the `base_model_name_or_path` is present in the [config](https://huggingface.co/ybelkada/opt-350m-lora/blob/main/adapter_config.json#L2)). Note that the adapter weights will still be separated, which you can see by printing the state dict:
```auto
for name, param in model.named_parameters():
print(name, param.shape)
```
which prints (among other things):
```auto
model.decoder.layers.23.self_attn.k_proj.weight torch.Size([1024, 1024])
model.decoder.layers.23.self_attn.k_proj.bias torch.Size([1024])
model.decoder.layers.23.self_attn.v_proj.base_layer.weight torch.Size([1024, 1024])
model.decoder.layers.23.self_attn.v_proj.base_layer.bias torch.Size([1024])
model.decoder.layers.23.self_attn.v_proj.lora_A.default.weight torch.Size([16, 1024])
model.decoder.layers.23.self_attn.v_proj.lora_B.default.weight torch.Size([1024, 16])
model.decoder.layers.23.self_attn.q_proj.base_layer.weight torch.Size([1024, 1024])
model.decoder.layers.23.self_attn.q_proj.base_layer.bias torch.Size([1024])
model.decoder.layers.23.self_attn.q_proj.lora_A.default.weight torch.Size([16, 1024])
model.decoder.layers.23.self_attn.q_proj.lora_B.default.weight torch.Size([1024, 16])
model.decoder.layers.23.self_attn.out_proj.weight torch.Size([1024, 1024])
model.decoder.layers.23.self_attn.out_proj.bias torch.Size([1024])
```
=\> so here you can clearly see that the adapter weights (lora\_A, lora\_B) are added to the query and values projection layers (base\_layer).
---
**Author:**  [@Rajneesh407](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRajneesh407)%5C
**Post date:** [December 30, 2023, 6:21am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-finetuning-a-lora-finetuned-causallm-model%2F36987%2F7 "2023-12-30T06:21:00Z")
Hi @nielsr ,
Thanks for writing this, it was very helpful.
Can you please extend this to how exactly we would load the step 2 fine tuned model for inferencing ? using PeftModel.from\_pretrained(some\_model, adapter\_2)
Now what exactly is this some model ?
a) The original model “facebook/opt-350m” or
b) it would be base\_with\_adapters\_model.merge\_and\_unload()
I am really confused about this. Any help would be really appreciated.
---
**Author:**  [@luispintoc](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fluispintoc)%5C
**Post date:** [March 14, 2024, 3:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-finetuning-a-lora-finetuned-causallm-model%2F36987%2F8 "2024-03-14T03:05:36Z")
Hi, could you please elaborate more on the second posibility:
> adding an adapter which you train on one dataset, then you train it on another dataset, etc. (so only 1 adapter). Eventually you could merge it into the base model
---
**Author:**  [@mayjul](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmayjul)%5C
**Post date:** [April 10, 2024, 11:37am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-finetuning-a-lora-finetuned-causallm-model%2F36987%2F10 "2024-04-10T11:37:50Z")
Hi, is there a way to implement the second possibility?
“adding an adapter which you train on one dataset, then you train it on another dataset, etc. (so only 1 adapter). Eventually you could merge it into the base model”
---
**Author:**  [@harry71](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fharry71)%5C
**Post date:** [April 16, 2024, 8:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-finetuning-a-lora-finetuned-causallm-model%2F36987%2F11 "2024-04-16T08:40:37Z")
Hi @luispintoc @mayjul
as per nielsr . if i get it correct you can finetune using LoRa on your custom datasets.
- Create a dataset for Task1 , you finetune it using LoRa, but you wont be saving this adapter yet.
- You will Again have a new dataset for Task2, you will then continue the finetuning on top of the previous finetuned weights,
- Can iteratively do this and once you are finished with different datasets you can then save the Adapter weights .
This Adapter weights should contain the knowledge from the different iterations you did on different datasets.
This is just a possibillity in which you can do , else you can combine different datasets together too and run it at once also.
Do correct me if you meant otherwise nielsr .
---
**Author:**  [@harry71](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fharry71)%5C
**Post date:** [April 16, 2024, 8:43am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-finetuning-a-lora-finetuned-causallm-model%2F36987%2F12 "2024-04-16T08:43:21Z")
Hi @JulianGerhard
The approach is fine and it does work. i had tried similar for my usecase, although I am still skeptical on how much the Finetuned adapters affect and influence the results with updated knowledge .
---
**Author:**  [@LeroyDyer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLeroyDyer)%5C
**Post date:** [May 2, 2024, 3:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-finetuning-a-lora-finetuned-causallm-model%2F36987%2F13 "2024-05-02T15:01:33Z")
Its quite interesting to read: - but how was model update Pre lora? / Peft models? and can they still be updated or fine tuned this way:
---
**Author:**  [@UserDAN](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FUserDAN)%5C
**Post date:** [May 3, 2024, 9:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-finetuning-a-lora-finetuned-causallm-model%2F36987%2F14 "2024-05-03T09:06:39Z")
Actually I have the same question as you I want to know how much affect has been done on the original base model because it seems to me when I evaluate the model it is still the same
---
**Author:**  [@LeroyDyer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLeroyDyer)%5C
**Post date:** [May 7, 2024, 9:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-finetuning-a-lora-finetuned-causallm-model%2F36987%2F15 "2024-05-07T09:55:21Z")
yes i think the loras are not great unless saved after the first pass ; so if the lora was stoped and started continually its worthless:
I also did a low rank decopostion on my model and added to the mistral base and it did not make much difference !: personally i merge all my loras with my model !:
SO:
To create a lora it should always be made from the mistral base ! not form a model on top of a model ! then the lora can be used as an add on 🙂
or use your own base model only (no merged) ; then the loras you make will have effect … but if you merge them your model already moved forwards from when the previous loras were created so adding them will send you back in time … only usefull if your latest training messed up … crazy ideas … and is doing mad stuff … you can use your loras to revery a little bit back in time !
so create a great base model 🙂
then create many loras for specific tasks (so you dont mess with your model ) … then use the loras as adapters: (for your model) when sharing they need to use your base ! unless you did not upgrade it from the original base?
BUT 🙂
If you upgrade you base then all loras previous ar invalid and wont effect you ! and may degrade the model to its previous bad responses!
---
**Author:**  [@dstnrsh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdstnrsh)%5C
**Post date:** [May 25, 2024, 3:12pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-finetuning-a-lora-finetuned-causallm-model%2F36987%2F16 "2024-05-25T15:12:47Z")
How do you stack loras or add multiple loras to a model and are they the same thing or is “stacking” something different?
---
**Author:**  [@UserDAN](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FUserDAN)%5C
**Post date:** [May 26, 2024, 8:43am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-finetuning-a-lora-finetuned-causallm-model%2F36987%2F17 "2024-05-26T08:43:30Z")
I believe it you are saying is very interesting but I’m not sure if I get you fully correctly I will really really appreciate it if you can give us a simple code of what do you mean by add as many Loras without merge
---
**Author:**  [@LeroyDyer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLeroyDyer)%5C
**Post date:** [July 7, 2024, 9:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-finetuning-a-lora-finetuned-causallm-model%2F36987%2F19 "2024-07-07T21:01:39Z")
sorry i will have to search for my code on this but here is an article very easy to follow i do remember taking my code from the original hugging face docs somewhere though:
> **[Combine Multiple LoRA Adapters for Llama 2](https://kaitchup.substack.com/p/combine-multiple-lora-adapters-for)**
>
> Add skills to your LLM without fine-tuning new adapters
---
**Author:**  [@LeroyDyer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLeroyDyer)%5C
**Post date:** [July 7, 2024, 9:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffurther-finetuning-a-lora-finetuned-causallm-model%2F36987%2F20 "2024-07-07T21:03:49Z")
you could add adapters to your model , so now you will have multiple loras loaded and you can switch between loras for specific querys, ie routing or you can even combine multiple adapters into a single adapter … hence creating a lora with combined abilitys(deltas) which can be further outputted to pretrained evenmerged with the model …