# Inference after QLoRA fine-tuning **URL:** **Author:** ![lewisbails](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewisbails/32/19003_2.png) [@lewisbails](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flewisbails)%5C **Post date:** [August 30, 2023, 8:32am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-after-qlora-fine-tuning%2F52772%2F1 "2023-08-30T08:32:10Z") I’ve fine-tuned a model via QLoRA by following this notebook: [Google Colab](https://colab.research.google.com/drive/1VoYNfYDKcKRQRor98Zbf2-9VQTtGJ24k?usp=sharing) And I pushed the adapter weights to the hugging face hub. When it comes time to predict with the base model+adapters, should I quantize the base model again (given the adapters were trained alongside a frozen quantized base model)? Or is it valid to load the base model unquantized, attach/merge the adapters as usual, and predict away? ```python ## TRAINING bnb_config = BitsAndBytesConfig(...) model = AutoModelForCausalLM.from_pretrained(..., quantization_config=bnb_config) # fit adapters alongside quantized base model model = prepare_model_for_kbit_training(model) peft_config = LoraConfig(...) trainer = SFTTrainer(model=model, peft_config=peft_config, ...) trainer.train() trainer.push_to_hub() # pushes adapter weights only ## INFERENCE peft_config = PeftConfig.from_pretrained() base_model = AutoModelForCausalLM.from_pretrained(peft_config.base_model_name_or_path, device_map="auto") # should I quantize here as I did when fitting the adapters with QLoRA? peft_model = PeftModel.from_pretrained(base_model, ) model = peft_model.merge_and_unload() ``` ---
**Author:** ![sade-adrien](https://avatars.discourse-cdn.com/v4/letter/s/8baadc/32.png) [@sade-adrien](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsade-adrien)%5C **Post date:** [October 3, 2023, 8:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-after-qlora-fine-tuning%2F52772%2F2 "2023-10-03T20:43:08Z")
This should help: > **[LoRA Adapters: When a Naive Merge Leads to Poor Performance](https://kaitchup.substack.com/p/lora-adapters-when-a-naive-merge?utm_source=%2Fsearch%2Fqlora&utm_medium=reader2)** > > The case of LoRA adapters fine-tuned with QLoRA ---
**Author:** ![datadoktergroen](https://avatars.discourse-cdn.com/v4/letter/d/c5a1d2/32.png) [@datadoktergroen](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdatadoktergroen)%5C **Post date:** [November 7, 2023, 4:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-after-qlora-fine-tuning%2F52772%2F3 "2023-11-07T16:31:14Z")
Unfortunately lots of the content is behind a paywall… Any other sources? ---
**Author:** ![sade-adrien](https://avatars.discourse-cdn.com/v4/letter/s/8baadc/32.png) [@sade-adrien](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsade-adrien)%5C **Post date:** [November 7, 2023, 5:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-after-qlora-fine-tuning%2F52772%2F4 "2023-11-07T17:05:55Z")
Apparently, PEFT v.0.6.0 which was released a couple of days ago is now able to to merge LoRA weights with 8bits base models. Should look more into it to make sure it’s done right, but if we have faith it should be fine. Haven’t tried it yet tho ---
**Author:** ![Owos](https://avatars.discourse-cdn.com/v4/letter/o/87869e/32.png) [@Owos](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FOwos)%5C **Post date:** [January 12, 2024, 11:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-after-qlora-fine-tuning%2F52772%2F5 "2024-01-12T23:52:30Z")
HI @lewisbails did you find an answer your question? I have the same question right now. ---
**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C **Post date:** [January 13, 2024, 10:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-after-qlora-fine-tuning%2F52772%2F6 "2024-01-13T10:46:07Z")
If you fine-tuned a model using PEFT, then at inference time you can just use the AutoModelForCausalLM class, which will automatically load the base model + adapter weights for you (thanks to the [PEFT integration](https://huggingface.co/docs/peft/tutorial/peft_integrations#transformers) in Transformers). You can additionally also pass `load_in_4bit=True` and `device_map="auto"` in order to do 4-bit inference and automatically place the model on the available GPUs. I’ve made a [notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Mistral/Supervised_fine_tuning_(SFT)_of_an_LLM_using_Hugging_Face_tooling.ipynb) that goes over QLoRa fine-tuning of Mistral-7B which also includes an inference section. ---
**Author:** ![mooolie](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@mooolie](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmooolie)%5C **Post date:** [April 4, 2024, 3:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-after-qlora-fine-tuning%2F52772%2F7 "2024-04-04T03:53:26Z")
you’re a real one brother. 😁 ---
**Author:** ![rohit103](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/rohit103/32/27848_2.png) [@rohit103](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frohit103)%5C **Post date:** [June 5, 2024, 9:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-after-qlora-fine-tuning%2F52772%2F8 "2024-06-05T21:03:18Z")
Hello Niels, Thank you for sharing. Did you ever try to compare QLoRA with QA-LoRA in terms of speed and accuracy during the inference time? I would greatly appreciate sharing your experiences. Best, Rohit ---
**Author:** ![Pavansatish](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pavansatish/32/27955_2.png) [@Pavansatish](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPavansatish)%5C **Post date:** [June 7, 2024, 3:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finference-after-qlora-fine-tuning%2F52772%2F9 "2024-06-07T15:11:55Z")
hey can you help me? There is an issue while running on my local Mac machine. When changing the model name from “microsoft/DialoGPT-small” to “meta-llama/Meta-Llama-3-8B”, the machine is hanging and no response. [GitHub Link](https://github.com/domschl/HuggingFaceGuidedTourForMac/blob/main/01-ChatBot.ipynb)