# Do\_sample defaults to False when I'm trying to set to True
**URL:**
**Author:**  [@anniedoris](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fanniedoris)%5C
**Post date:** [June 10, 2024, 7:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdo-sample-defaults-to-false-when-im-trying-to-set-to-true%2F91380%2F1 "2024-06-10T19:07:34Z")
Hi,
I am trying to run inference on the model lmsys/vicuna-7b-v1.5. I load the model and set up the pipeline/prompting as follows:
```auto
tokenizer = AutoTokenizer.from_pretrained(model_name)
model_pipeline = transformers.pipeline(
"text-generation",
model=model_name,
tokenizer = tokenizer,
torch_dtype=torch.float16,
max_new_tokens = max_new_toks,
device_map="auto"
)
# There are other traditional LLM inference settings that can be modified here
sequences = model_pipeline(
input_prompt,
do_sample=True
)
```
Even though I set do\_sample as True, I get numerous warnings that say:
```auto
UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.
```
Am I doing something incorrectly when setting do\_sample? When the model is loaded, does it set all inference parameters to those that are in the model’s associated generation\_config.json file? How would you recommend resolving this? Thank you!
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [June 10, 2024, 7:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdo-sample-defaults-to-false-when-im-trying-to-set-to-true%2F91380%2F2 "2024-06-10T19:17:36Z")
Hi,
Thanks for reporting, that looks like a bug. It might have to do with the generation config on the hub: [generation\_config.json · lmsys/vicuna-7b-v1.5 at main](https://huggingface.co/lmsys/vicuna-7b-v1.5/blob/main/generation_config.json).
cc @joaogante
---
**Author:**  [@RaushanTurganbay](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRaushanTurganbay)%5C
**Post date:** [June 11, 2024, 5:16am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdo-sample-defaults-to-false-when-im-trying-to-set-to-true%2F91380%2F3 "2024-06-11T05:16:02Z")
Indeed, the warning is printed when loading the generation config from the hub, so can be avoided by modifying config directly on the hub.
The generation in this code snippet is done with sampling, as indicated by `do_sample=True`
---
**Author:**  [@anniedoris](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fanniedoris)%5C
**Post date:** [June 13, 2024, 2:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdo-sample-defaults-to-false-when-im-trying-to-set-to-true%2F91380%2F4 "2024-06-13T14:34:45Z")
Thanks for the responses. How would I be able to modify the generation config on the hub? It also doesn’t look like the generation config on the hub has any mentions of `do_sample`?
---
**Author:**  [@RaushanTurganbay](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRaushanTurganbay)%5C
**Post date:** [June 14, 2024, 5:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdo-sample-defaults-to-false-when-im-trying-to-set-to-true%2F91380%2F5 "2024-06-14T05:34:33Z")
I think the only way to open a PR in the hub asking the owners to add `do_sample=True` in the generation config file. Currently it doesn’t have any mention of `do_sample` and the default value is False in that case
---
**Author:**  [@Chahnwoo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FChahnwoo)%5C
**Post date:** [June 18, 2024, 12:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdo-sample-defaults-to-false-when-im-trying-to-set-to-true%2F91380%2F6 "2024-06-18T00:06:36Z")
@anniedoris I think the problem might be how you’re passing the keyword argument to the pipeline. Instead of setting `do_sample=True`, try `{ "do_sample" : True }`