# Combine between lora and prompt tunning

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcombine-between-lora-and-prompt-tunning%2F65151%3E%5C
**Category:** 🤗Transformers\
**Created:** [December 9, 2023, 10:22am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcombine-between-lora-and-prompt-tunning%2F65151 "2023-12-09T10:22:55Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![hazemZ](https://avatars.discourse-cdn.com/v4/letter/h/e19b73/32.png) [@hazemZ](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FhazemZ)%5C
**Post date:** [December 9, 2023, 10:22am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcombine-between-lora-and-prompt-tunning%2F65151%2F1 "2023-12-09T10:22:55Z")

</div>

Iam trying to fine tunne LLM using prompt tunning and lora by combining them and start training

1-I freezed both model weights and embedding parameters so i used this :

```auto
# freeze the model - train adapters later
for param in model.parameters():
  param.requires_grad = False
  if param.ndim == 1:
    # cast the small parameters (e.g. layernorm) to fp32 for stability
    param.data = param.data.to(torch.float32)

# Freeze original embedding layer
for param in model.config.embeddings.encoder.parameters():
    param.requires_grad = False

model.enable_input_require_grads()

```

2- I intialized prompt configuration and enjected it with the model:

```auto
from peft import PromptTuningConfig, LoraConfig, TaskType, PromptTuningInit, get_peft_model

# Create the PromptTuningConfig object for quiz generation
prompt_config = PromptTuningConfig(
    task_type=TaskType.CAUSAL_LM,
    num_virtual_tokens=10,
    prompt_tuning_init=PromptTuningInit.TEXT,
    prompt_tuning_init_text="", # what your model for: 
    tokenizer_name_or_path=tokenizer
)

model = get_peft_model(model, prompt_config)

```

3- also intialized the lora configuration:

```auto
Lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q", "v"],
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.SEQ_2_SEQ_LM
)

model = get_peft_model(model, Lora_config)

```

4-then i start training

```auto
import transformers

trainer = transformers.Trainer(
    model=model,
    train_dataset="", # your training data
    args=transformers.TrainingArguments(
        per_device_train_batch_size=8,
        gradient_accumulation_steps=8,
        warmup_steps=100,
        max_steps=1000,
        learning_rate=1e-3,
        fp16=True,
        logging_steps=1,
        output_dir='outputs',
    ),
    data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False)
)
model.config.use_cache = False # silence the warnings. Please re-enable for inference!
trainer.train()

```

at the end all i need to khnow is :

1-Are the additional prompt tokens updated or not , i khnow that the lora wieghts or matrix got updated iam not sure of the prompt tokens

2- since model original weights in lora kept frozen during training , Are the original model embeddings kept frozen too or got changed ?

---

<div class="post-metadata">

**Author:** ![ambuje](https://avatars.discourse-cdn.com/v4/letter/a/cab0a1/32.png) [@ambuje](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fambuje)%5C
**Post date:** [February 3, 2024, 12:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcombine-between-lora-and-prompt-tunning%2F65151%2F2 "2024-02-03T12:35:24Z")

</div>

Where you able to verify? and can you also share some results regarding combining prompt tuning + LORA?

Thank you,