# Setting up my custom device map for a LLM
**URL:**
**Author:**  [@SanjuEpic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSanjuEpic)%5C
**Post date:** [June 30, 2023, 11:35am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsetting-up-my-custom-device-map-for-a-llm%2F45132%2F1 "2023-06-30T11:35:04Z")
Hello community, greetings!
This is my first post on forums, and i have a quick doubt on
“”"let’s say i want to load bigscience/bloom-1b7 model, and i have just enough GPU RAM to fit the entire model except the lm\_head. Therefore write a custom device\_map as follows:
device\_map = {
“transformer.word\_embeddings”: 0,
“transformer.word\_embeddings\_layernorm”: 0,
“lm\_head”: “cpu”,
“transformer.h”: 0,
“transformer.ln\_f”: 0,
}
“”"
This part has been taken from here: [Quantize 🤗 Transformers models (huggingface.co)](https://huggingface.co/docs/transformers/main_classes/quantization#offload-between-cpu-and-gpu)
How can one know the architecture of that particular model before actually trying to setup a custom device\_map.
If i want to try help(model) to get the architecture of it, in general i wont be able to load this model in the first hand on a GPU, if this is the case then how can i know which particular layers i can offload between cpu and gpu to quantize it?
Let me know if there is any adequacy in my question.
Thanks!
---
**Author:**  [@DesmondChoy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDesmondChoy)%5C
**Post date:** [August 10, 2023, 1:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsetting-up-my-custom-device-map-for-a-llm%2F45132%2F2 "2023-08-10T13:01:06Z")
Yeah that’s a question that I’ve been wondering about too - how do you find the layers for each model? I looked up [bigscience/bloom-1b7 · Hugging Face](https://huggingface.co/bigscience/bloom-1b7) and searched for “lm\_head” and wasn’t able to find any results.
---
**Author:**  [@SanjuEpic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSanjuEpic)%5C
**Post date:** [August 10, 2023, 3:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsetting-up-my-custom-device-map-for-a-llm%2F45132%2F3 "2023-08-10T15:22:46Z")
One naive solution I found out was to get the device map of that model by running it on a larger gpu machine and store it somewhere and later on use that device map according to the smaller gpu architecture for cpu and gpu offloading.
model.hf\_device\_map function will help for knowing the device map of a hugging face model
---
**Author:**  [@nitinsurya](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnitinsurya)%5C
**Post date:** [January 29, 2025, 1:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsetting-up-my-custom-device-map-for-a-llm%2F45132%2F4 "2025-01-29T13:24:22Z")
This post has some ideas: [Error: The model weights are not tied. Please use the `tie_weights` method before using the `infer_auto_device` function, even after adding model.tie\_weights()](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferror-the-model-weights-are-not-tied-please-use-the-tie-weights-method-before-using-the-infer-auto-device-function-even-after-adding-model-tie-weights%2F46325)
Maybe this could work:
```auto
import transformers
from accelerate import init_empty_weights
from transformers import AutoConfig, AutoModelForCausalLM
config = AutoConfig.from_pretrained("chaoyi-wu/PMC_LLAMA_7B")
with init_empty_weights():
model = AutoModelForCausalLM.from_config(config)
from accelerate import infer_auto_device_map, init_empty_weights
device_map = infer_auto_device_map(model)
```