# How to set model\_path to another directory?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-set-model-path-to-another-directory%2F34690%3E%5C
**Category:** Beginners\
**Created:** [March 26, 2023, 7:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-set-model-path-to-another-directory%2F34690 "2023-03-26T07:17:37Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![TheAIGuyz](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/theaiguyz/32/14944_2.png) [@TheAIGuyz](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTheAIGuyz)%5C
**Post date:** [March 26, 2023, 7:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-set-model-path-to-another-directory%2F34690%2F1 "2023-03-26T07:17:37Z")

</div>

Hello! I am trying to follow these instructions to have both GPU and CPU offloading  
[https://huggingface.co/docs/transformers/main/en/main\_classes/quantization#offload-between-cpu-and-gpu](https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu)

Below is the code I am using

# Set the quantization config with llm\_int8\_enable\_fp32\_cpu\_offload set to True

quantization\_config = BitsAndBytesConfig(llm\_int8\_enable\_fp32\_cpu\_offload=True)

device\_map = {  
“transformer.word\_embeddings”: 0,  
“transformer.word\_embeddings\_layernorm”: 0,  
“lm\_head”: “cpu”,  
“transformer.h”: 0,  
“transformer.ln\_f”: 0,  
}

model\_path = “decapoda-research/llama-7b-hf”  
model\_8bit = AutoModelForCausalLM.from\_pretrained(  
model\_path,  
device\_map=device\_map,  
quantization\_config=quantization\_config,  
)

my problem is that it asks for the model\_path to link to hugging face and a specific model. I also have these models downloaded but in another path:  
C:\Windows\System32\text-generation-webui\models

When I run the code above it downloads the model again. How do I get the model\_path to look at my already downloaded models?

Even when I let the download run, I get this error

Traceback (most recent call last):  
File “C:\Windows\System32\text-generation-webui\server7b.py”, line 33, in   
model\_8bit = AutoModelForCausalLM.from\_pretrained(  
File “C:\Users\justi\miniconda3\envs\textgen\lib\site-packages\transformers\models\auto\auto\_factory.py”, line 471, in from\_pretrained  
return model\_class.from\_pretrained(  
File “C:\Users\justi\miniconda3\envs\textgen\lib\site-packages\transformers\modeling\_utils.py”, line 2643, in from\_pretrained  
) = cls.\_load\_pretrained\_model(  
File “C:\Users\justi\miniconda3\envs\textgen\lib\site-packages\transformers\modeling\_utils.py”, line 2966, in \_load\_pretrained\_model  
new\_error\_msgs, offload\_index, state\_dict\_index = \_load\_state\_dict\_into\_meta\_model(  
File “C:\Users\justi\miniconda3\envs\textgen\lib\site-packages\transformers\modeling\_utils.py”, line 662, in \_load\_state\_dict\_into\_meta\_model  
raise ValueError(f"{param\_name} doesn’t have any device set.")  
ValueError: model.layers.0.self\_attn.q\_proj.weight doesn’t have any device set.

---

<div class="post-metadata">

**Author:** ![Briannab3d](https://avatars.discourse-cdn.com/v4/letter/b/c68b51/32.png) [@Briannab3d](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBriannab3d)%5C
**Post date:** [June 12, 2023, 5:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-set-model-path-to-another-directory%2F34690%2F2 "2023-06-12T17:49:16Z")

</div>

> [@TheAIGuyz](#):
>
> File “C:\Users\justi\miniconda3\envs\textgen\lib\site-packages\transformers\modeling\_utils.py”, line 662, in \_load\_state\_dict\_into\_meta\_model  
> raise ValueError(f"{param\_name} doesn’t have any device set.")  
> ValueError: model.layers.0.self\_attn.q\_proj.weight doesn’t have any device set.

looks like your devicemap isn’t covering something called model.layers.0.self\_attn.q\_proj.weight  
Lets make a net maybe

# Create the device map dictionary with default device assignment to catch anything not specified

device\_map = {param.name: default\_device for param in model.parameters()}

Assign specific devices to parameters  
device\_map.update({  
“transformer.word\_embeddings”: torch.device(“cuda:0”),  
“transformer.word\_embeddings\_layernorm”: torch.device(“cuda:0”),  
“lm\_head”: torch.device(“cpu”),  
“transformer.h”: torch.device(“cuda:0”),  
“transformer.ln\_f”: torch.device(“cuda:0”)  
})

## if you want to get specific,

maybe something like this::  
from transformers import AutoModelForCausalLM  
model\_path = “decapoda-research/llama-7b-hf”  
model = AutoModelForCausalLM.from\_pretrained(model\_path, local\_files\_only=True)

## Get the names of the model parameters

param\_names = [param.name for param in model.parameters()]

### Print the parameter names

for name in param\_names:  
print(name)  
iterating over `model.parameters()` to find each parameter and retrieve its `name` attribute. This will provide you with the names of the model’s parameters for device assignment

> [@TheAIGuyz](#):
>
> When I run the code above it downloads the model again. How do I get the model\_path to look at my already downloaded models?

# Maybe have it check for the file first:

import os  
from transformers import AutoModelForCausalLM  
model\_name = “llama-7b-hf”  
model\_directory = “models”

## Check if the model file exists in the specified directory

model\_file = os.path.join(model\_directory, model\_name)  
if os.path.exists(model\_file):  
model\_path = model\_file  
print(“Model file found in the directory. Using the local model file.”)  
else:  
model\_path = model\_name  
print(“Model file not found in the directory. Downloading the model from the repository.”)

## Load the model

model = AutoModelForCausalLM.from\_pretrained(model\_path)