# Fine Tune with/without LORA

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-with-without-lora%2F100477%3E%5C
**Category:** 🤗Transformers\
**Created:** [August 1, 2024, 9:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-with-without-lora%2F100477 "2024-08-01T21:32:57Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![bochen0909](https://avatars.discourse-cdn.com/v4/letter/b/df788c/32.png) [@bochen0909](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbochen0909)%5C
**Post date:** [August 1, 2024, 9:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-with-without-lora%2F100477%2F1 "2024-08-01T21:32:57Z")

</div>

Fine tune a 8G model on chat data with/without LORA.

With LORA:

```auto
peft_config = LoraConfig(
        lora_alpha=128,
        lora_dropout=0.05,
        r=256,
        bias="none",
        target_modules="all-linear",
        task_type="CAUSAL_LM",
    )
trainer = SFTTrainer(
        model=model,
        args=training_args,
        train_dataset=dataset,
        peft_config=peft_config,
        max_seq_length=max_seq_length,
        tokenizer=tokenizer,
        packing=True,
        dataset_kwargs={
            "add_special_tokens": False,
            "append_concat_token": False,
        },
    )

```

Without LORA:

```auto
    trainer = SFTTrainer(
        model=model,
        args=training_args,
        train_dataset=dataset,
        max_seq_length=max_seq_length,
        tokenizer=tokenizer,
        packing=True,
        dataset_kwargs={
            "add_special_tokens": False,
            "append_concat_token": False,
        },
    )

```

All other things and parameters are same. LORA gave really good results, without-LORA produced non related response.

Anyone has experience and clues on the reasons?

---

<div class="post-metadata">

**Author:** ![huggingzob](https://avatars.discourse-cdn.com/v4/letter/h/b3f665/32.png) [@huggingzob](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhuggingzob)%5C
**Post date:** [October 7, 2024, 7:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-with-without-lora%2F100477%2F2 "2024-10-07T19:39:30Z")

</div>

Isn’t it the expected result? LORA would be easier to train so if you are using same number of epochs LORA should learn faster I believe.