# Help with Llama 2 Finetuning Setup **URL:** **Author:** ![nathng](https://avatars.discourse-cdn.com/v4/letter/n/ad7895/32.png) [@nathng](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnathng)%5C **Post date:** [August 8, 2023, 10:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-llama-2-finetuning-setup%2F50035%2F1 "2023-08-08T22:14:36Z") Hello! I’m trying to follow the Llama 2 finetunning example provided by databricks here [https://github.com/databricks/databricks-ml-examples/blob/b1ca47c058461f7fde214914d53b051990064d94/llm-models/llamav2/llamav2-7b/scripts/fine\_tune\_deepspeed.py#L94](https://github.com/databricks/databricks-ml-examples/blob/b1ca47c058461f7fde214914d53b051990064d94/llm-models/llamav2/llamav2-7b/scripts/fine_tune_deepspeed.py#L94) . However, I ran into the following issue “lib/python3.9/site-packages/transformers/generation/configuration\_utils.py”, line 354, in validate raise ValueError( ValueError: `do_sample` is set to `False`. However, temperature is set to 0.9 – this flag is only used in sample-based generation modes. Set `do_sample=True` or unset temperature to continue." on the " model = transformers.AutoModelForCausalLM.from\_pretrained( pretrained\_model\_name\_or\_path, torch\_dtype=torch.float16, trust\_remote\_code=True, use\_auth\_token=True, )" line for the meta-llama/Llama-2-7b-chat-hf model. I don’t set temperature anywhere in the script. Does anyone have any idea what the issue is? ---
**Author:** ![vijayjadhavaug](https://avatars.discourse-cdn.com/v4/letter/v/73ab20/32.png) [@vijayjadhavaug](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvijayjadhavaug)%5C **Post date:** [August 9, 2023, 4:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-llama-2-finetuning-setup%2F50035%2F2 "2023-08-09T04:11:37Z")
try setting the temperature parameter to 0.1 while initialising the hugging face pipeline ---
**Author:** ![muzammil-eds](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muzammil-eds/32/18385_2.png) [@muzammil-eds](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuzammil-eds)%5C **Post date:** [August 9, 2023, 6:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-llama-2-finetuning-setup%2F50035%2F3 "2023-08-09T06:01:26Z")
ValueError: `do_sample` is set to `False`. However, temperature is set to 0.9 – this flag is only used in sample-based generation modes. Set `do_sample=True` or unset temperature to continue. my code is: from transformers import GenerationConfig, LlamaForCausalLM, LlamaTokenizer MODEL\_NAME = “meta-llama/Llama-2-7b-chat-hf” bnb\_config = BitsAndBytesConfig( load\_in\_4bit=True, bnb\_4bit\_use\_double\_quant=True, bnb\_4bit\_quant\_type=“nf4”, bnb\_4bit\_compute\_dtype=torch.bfloat16, ) model = LlamaForCausalLM.from\_pretrained( MODEL\_NAME, device\_map=“auto”, trust\_remote\_code=True, use\_auth\_token=True, temperature=0.1, do\_sample=True, quantization\_config=bnb\_config, ) tokenizer = LlamaTokenizer.from\_pretrained(MODEL\_NAME) tokenizer.pad\_token = tokenizer.eos\_token i explicitly change the do\_sample=True in configuration\_utils.py but didn`t worked ---
**Author:** ![Monlp](https://avatars.discourse-cdn.com/v4/letter/m/bc8723/32.png) [@Monlp](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMonlp)%5C **Post date:** [August 9, 2023, 6:44am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-llama-2-finetuning-setup%2F50035%2F4 "2023-08-09T06:44:06Z")
I am having the same issue. It was working without problem until last night. I tried to change the config file and update it by adding do\_sample=true but did not work. !pip install -qqq bitsandbytes --progress-bar off !pip install -qqq torch --progress-bar off !pip install -q -U git+https://github.com/huggingface/transformers.git MODEL\_NAME = “meta-llama/Llama-2-7b-chat-hf” model = AutoModelForCausalLM.from\_pretrained(MODEL\_NAME, device\_map=“auto”, quantization\_config=bnb\_config) Error msg: ValueError Traceback (most recent call last) in \() ----\> 1 model = AutoModelForCausalLM.from\_pretrained(MODEL\_NAME, device\_map=“auto”, quantization\_config=bnb\_config) 5 frames /usr/local/lib/python3.10/dist-packages/transformers/generation/configuration\_utils.py in validate(self) 352 ) 353 if self.temperature != 1.0: → 354 raise ValueError( 355 greedy\_wrong\_parameter\_msg.format(flag\_name=“temperature”, flag\_value=self.temperature) 356 ) ValueError: `do_sample` is set to `False`. However, temperature is set to 0.9 – this flag is only used in sample-based generation modes. Set `do_sample=True` or unset temperature to continue. ---
**Author:** ![nathng](https://avatars.discourse-cdn.com/v4/letter/n/ad7895/32.png) [@nathng](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnathng)%5C **Post date:** [August 9, 2023, 6:57am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-llama-2-finetuning-setup%2F50035%2F5 "2023-08-09T06:57:14Z")
For anyone looking for a solution, it was an issue with the latest release of hugging face transformers released recently. Please downgrade to the previous version !pip install git+https://github.com/huggingface/transformers@v4.31-release to fix the issue. ---
**Author:** ![Monlp](https://avatars.discourse-cdn.com/v4/letter/m/bc8723/32.png) [@Monlp](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMonlp)%5C **Post date:** [August 9, 2023, 7:08am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-llama-2-finetuning-setup%2F50035%2F6 "2023-08-09T07:08:34Z")
Thank you. Downgrading made it work 👍 ---
**Author:** ![muzammil-eds](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muzammil-eds/32/18385_2.png) [@muzammil-eds](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuzammil-eds)%5C **Post date:** [August 9, 2023, 7:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-llama-2-finetuning-setup%2F50035%2F7 "2023-08-09T07:19:02Z")
Thanks It works ---
**Author:** ![prajwalk](https://avatars.discourse-cdn.com/v4/letter/p/9dc877/32.png) [@prajwalk](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fprajwalk)%5C **Post date:** [August 9, 2023, 12:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-llama-2-finetuning-setup%2F50035%2F8 "2023-08-09T12:14:52Z")
I was using autotrain and got the same error but downgrading transformers didn’t solve the issue. ```auto !pip install huggingface_hub autotrain-advanced !pip install git+https://github.com/huggingface/transformers@v4.31-release ``` This is how I installed the package, Any solution for this ---
**Author:** ![Osd111](https://avatars.discourse-cdn.com/v4/letter/o/b3f665/32.png) [@Osd111](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FOsd111)%5C **Post date:** [January 30, 2024, 6:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-llama-2-finetuning-setup%2F50035%2F9 "2024-01-30T06:58:51Z")
besides downgrade any other solution? ---
**Author:** ![Osd111](https://avatars.discourse-cdn.com/v4/letter/o/b3f665/32.png) [@Osd111](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FOsd111)%5C **Post date:** [January 30, 2024, 7:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-llama-2-finetuning-setup%2F50035%2F10 "2024-01-30T07:26:48Z")
As the error raised on ```auto File "/usr/local/lib/python3.10/dist-packages/transformers/generation/configuration_utils.py", line 560, in save_pretrained raise ValueError( ValueError: The generation config instance is invalid -- `.validate()` throws warnings and/or exceptions. Fix these issues to save the configuration. ``` I just comment the code that raised the exception. ---
**Author:** ![Thinkcru](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thinkcru/32/13469_2.png) [@Thinkcru](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FThinkcru)%5C **Post date:** [January 30, 2024, 9:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-llama-2-finetuning-setup%2F50035%2F11 "2024-01-30T21:24:05Z")
I’m hitting the same error, doesn’t 🤗 do regression testing? This is code is so brittle, I just tried with release version `4.38.0.dev0` and it still is crapping out. ---
**Author:** ![pham-llm](https://avatars.discourse-cdn.com/v4/letter/p/898d66/32.png) [@pham-llm](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpham-llm)%5C **Post date:** [February 16, 2024, 7:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-llama-2-finetuning-setup%2F50035%2F12 "2024-02-16T19:15:01Z")
Downgrading to 4.31 did not solve the issue. Can anyone point me to the fix? ---
**Author:** ![NeerajSantosh](https://avatars.discourse-cdn.com/v4/letter/n/0ea827/32.png) [@NeerajSantosh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNeerajSantosh)%5C **Post date:** [February 27, 2024, 8:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-llama-2-finetuning-setup%2F50035%2F13 "2024-02-27T20:49:26Z")
This solution works. Thank you so much!! ---
**Author:** ![remobor1](https://avatars.discourse-cdn.com/v4/letter/r/a88e57/32.png) [@remobor1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fremobor1)%5C **Post date:** [March 18, 2024, 5:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-llama-2-finetuning-setup%2F50035%2F14 "2024-03-18T17:25:01Z")
I am at version 4.37.2 and I still see the issue. Can anyone let me know, please, if the issue also appears in most recent releases? Is there any fix on its way? I tried all suggestions I could find on this issue, none of them worked. Thanks ---
**Author:** ![cvetanovskaa](https://avatars.discourse-cdn.com/v4/letter/c/e95f7d/32.png) [@cvetanovskaa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcvetanovskaa)%5C **Post date:** [March 20, 2024, 7:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-llama-2-finetuning-setup%2F50035%2F15 "2024-03-20T19:54:58Z")
The latest transformers version I could get actually working & saving was 4.36.2 (it still shows the warning but doesn’t crash). Downgrading more than that caused error with other things (like not knowing what “mistral” is a model for example). ---
**Author:** ![yananchen](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/yananchen/32/37864_2.png) [@yananchen](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fyananchen)%5C **Post date:** [April 2, 2024, 2:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-llama-2-finetuning-setup%2F50035%2F16 "2024-04-02T02:26:57Z")
version 4.39.2 crashes due to this error. anybody encounters this ? ---
**Author:** ![tiago-machado](https://avatars.discourse-cdn.com/v4/letter/t/58956e/32.png) [@tiago-machado](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftiago-machado)%5C **Post date:** [May 20, 2024, 7:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-llama-2-finetuning-setup%2F50035%2F17 "2024-05-20T19:15:55Z")
I’m on version 4.40.1 and it still gives the same error. Downgrading doesn’t work for me since it crashes other parts of the code.