# How to use gated model in inference
**URL:**
**Author:**  [@delving-deep](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdelving-deep)%5C
**Post date:** [September 26, 2024, 11:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-gated-model-in-inference%2F108911%2F1 "2024-09-26T23:10:25Z")
Hi, I have obtained access to Meta llama3 models, and I am trying to use it for inference using the sample code from model card. When I run my inference script, it gives me an error 'Cannot access gated repo for url `huggingface..../meta-llama.....config.json`
So my question is how can I access this model from my inference script? Do I need to pass any authientication/api/token key to ensure my script can access the model?
Also how to get huggingface version of the model locally?
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [September 26, 2024, 11:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-gated-model-in-inference%2F108911%2F2 "2024-09-26T23:27:38Z")
You could do it by making the code look like this.
> **[meta-llama/Llama-3.1-8B-Instruct · Hugging Face](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
## Sample code
```py
import transformers
import torch
model_id = "meta-llama/Meta-Llama-3.1-8B-Instruct"
pipeline = transformers.pipeline(
"text-generation",
model=model_id,
model_kwargs={"torch_dtype": torch.bfloat16},
device_map="auto",
)
messages = [
{"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
{"role": "user", "content": "Who are you?"},
]
outputs = pipeline(
messages,
max_new_tokens=256,
)
print(outputs[0]["generated_text"][-1])
```
## Actual Code
```py
import transformers
import torch
hf_token = "hf_ *********" # When uploading code, never write directly!
model_id = "meta-llama/Meta-Llama-3.1-8B-Instruct"
pipeline = transformers.pipeline(
"text-generation",
model=model_id,
model_kwargs={"torch_dtype": torch.bfloat16},
device_map="auto",
token=hf_token,
)
messages = [
{"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
{"role": "user", "content": "Who are you?"},
]
outputs = pipeline(
messages,
max_new_tokens=256,
)
print(outputs[0]["generated_text"][-1])
```
You can make as many tokens as you want here. You should name your tokens in such a way that you can easily identify which is which.
[https://huggingface.co/settings/tokens](https://huggingface.co/settings/tokens)
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [September 27, 2024, 6:36am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-gated-model-in-inference%2F108911%2F3 "2024-09-27T06:36:20Z")
Make sure to authenticate with `huggingface-cli login` in the terminal before running the script.
---
**Author:**  [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [September 30, 2024, 9:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-gated-model-in-inference%2F108911%2F4 "2024-09-30T21:42:57Z")
Closed on September 30, 2024, 9:42pm UTC
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.