# Loading BloomForCausalLM from sharded checkpoints

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-bloomforcausallm-from-sharded-checkpoints%2F32201%3E%5C
**Category:** 🤗Accelerate\
**Created:** [February 21, 2023, 4:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-bloomforcausallm-from-sharded-checkpoints%2F32201 "2023-02-21T16:24:22Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gillner](https://avatars.discourse-cdn.com/v4/letter/g/7feea3/32.png) [@Gillner](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGillner)%5C
**Post date:** [February 21, 2023, 4:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-bloomforcausallm-from-sharded-checkpoints%2F32201%2F1 "2023-02-21T16:24:22Z")

</div>

Hey everyone,

I am currently working on my master thesis and have used the Transformers library succesfully for most of the experiments I wanted to conduct. The only thing I am stuck with is loading a sharded version of Bloom-7b1, which I am loading using the accelerate load\_checkpoint\_and\_dispatch following the guide [Big model inference](https://huggingface.co/docs/accelerate/usage_guides/big_modeling), which results in the following error.

> File “load\_bloom\_test.py”, line 22, in   
> model = load\_checkpoint\_and\_dispatch(  
> File “/opt/conda/lib/python3.8/site-packages/accelerate/big\_modeling.py”, line 375, in load\_checkpoint\_and\_dispatch  
> load\_checkpoint\_in\_model(  
> File “/opt/conda/lib/python3.8/site-packages/accelerate/utils/modeling.py”, line 699, in load\_checkpoint\_in\_model  
> set\_module\_tensor\_to\_device(model, param\_name, param\_device, value=param)  
> File “/opt/conda/lib/python3.8/site-packages/accelerate/utils/modeling.py”, line 105, in set\_module\_tensor\_to\_device  
> new\_module = getattr(module, split)  
> File “/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1260, in **getattr**  
> raise AttributeError(“‘{}’ object has no attribute ‘{}’”.format(  
> AttributeError: ‘BloomForCausalLM’ object has no attribute ‘word\_embeddings’

The code I am using works fine for other models, such as Galactica (for all variants, including galactica-120b) and looks like this:

```auto
import torch

from transformers import AutoTokenizer, AutoConfig, AutoModelForCausalLM
from accelerate import init_empty_weights, infer_auto_device_map, load_checkpoint_and_dispatch

# Loading model from config on meta device and using sharded checkpoints
model_name = "models/bloom-7b1"
config = AutoConfig.from_pretrained(model_name)
with init_empty_weights():
    model = AutoModelForCausalLM.from_config(config)

# Calculating device map
device_map = infer_auto_device_map(
    model, 
    no_split_module_classes=["BloomBlock"],
    dtype=torch.float16
)

# Loading the checkpoint according to auto device_map
model = load_checkpoint_and_dispatch(
    model,
    model_name,
    device_map=device_map
)

tokenizer = AutoTokenizer.from_pretrained(model_name)
inputs = tokenizer("Hello, my name is", return_tensors="pt")
inputs = inputs.to(0)
output = model.generate(inputs["input_ids"])
print(tokenizer.decode(output[0]))

```

I have redownloaded the model a couple of times and also tried using `bigscience/bloom-7b1` to download it directly from the hub during execution of the script.

I would like this approach to work, as it has worked fine for all other models and I have not managed to get a distributed setup working using a different approach when using a model that doesn’t fit on a single GPU anymore.

Any help with either using a different loading approach or a solution to my problem would be greatly appreciated

---

<div class="post-metadata">

**Author:** ![sh0416](https://avatars.discourse-cdn.com/v4/letter/s/dc4da7/32.png) [@sh0416](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsh0416)%5C
**Post date:** [March 7, 2023, 3:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-bloomforcausallm-from-sharded-checkpoints%2F32201%2F2 "2023-03-07T15:14:34Z")

</div>

I also get and error, but slightly different.

```auto
Traceback (most recent call last):
  File "run_inference.py", line 12, in <module>
    model = load_checkpoint_and_dispatch(
  File "/home/anaconda3/envs/research/lib/python3.8/site-packages/accelerate/big_modeling.py", line 427, in load_checkpoint_and_dispatch
    load_checkpoint_in_model(
  File "/home/anaconda3/envs/research/lib/python3.8/site-packages/accelerate/utils/modeling.py", line 748, in load_checkpoint_in_model
    raise ValueError(f"{param_name} doesn't have any device set.")
ValueError: word_embeddings.weight doesn't have any device set.

```

---

<div class="post-metadata">

**Author:** ![sh0416](https://avatars.discourse-cdn.com/v4/letter/s/dc4da7/32.png) [@sh0416](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsh0416)%5C
**Post date:** [March 7, 2023, 3:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-bloomforcausallm-from-sharded-checkpoints%2F32201%2F3 "2023-03-07T15:19:20Z")

</div>

The same code for gpt-j-6B works.

Reference: [Handling big models for inference](https://huggingface.co/docs/accelerate/usage_guides/big_modeling)

@sgugger Could you check it for us?

Also, OPT raises an error.

```auto
Traceback (most recent call last):
  File "run_inference.py", line 13, in <module>
    model = load_checkpoint_and_dispatch(
  File "/home/anaconda3/envs/research/lib/python3.8/site-packages/accelerate/big_modeling.py", line 427, in load_checkpoint_and_dispatch
    load_checkpoint_in_model(
  File "/home/anaconda3/envs/research/lib/python3.8/site-packages/accelerate/utils/modeling.py", line 748, in load_checkpoint_in_model
    raise ValueError(f"{param_name} doesn't have any device set.")
ValueError: decoder.embed_tokens.weight doesn't have any device set.

```

The code I used it for testing.

```auto
from accelerate import init_empty_weights, load_checkpoint_and_dispatch
from transformers import AutoConfig, AutoModelForCausalLM

# checkpoint = "bigscience/bloom-7b1"
# checkpoint = "EleutherAI/gpt-j-6B"
checkpoint = "facebook/opt-6.7b"
config = AutoConfig.from_pretrained(checkpoint)

with init_empty_weights():
    model = AutoModelForCausalLM.from_config(config)

model = load_checkpoint_and_dispatch(
    model,
    # "bloom-7b1",
    # "sharded-gpt-j-6B",
    "opt-6.7b",
    device_map="sequential",
    # no_split_module_classes=["BloomBlock"],
    # no_split_module_classes=["GPTJBlock"],
    no_split_module_classes=["OPTDecoderLayer"],
)

print(model.hf_device_map)

```

I tried using different `device_map`, but it didn’t work…

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [March 7, 2023, 5:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-bloomforcausallm-from-sharded-checkpoints%2F32201%2F4 "2023-03-07T17:15:05Z")

</div>

You won’t be able to use the checkpoint on the Hub directly in Accelerate, you need to pass via Transformer, since the model on the Hub is the checkpoint of the base model, so not the model with the LM head.

---

<div class="post-metadata">

**Author:** ![sh0416](https://avatars.discourse-cdn.com/v4/letter/s/dc4da7/32.png) [@sh0416](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsh0416)%5C
**Post date:** [March 8, 2023, 1:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-bloomforcausallm-from-sharded-checkpoints%2F32201%2F5 "2023-03-08T01:50:04Z")

</div>

Could I ask what is the “Transformer” you mentioned? Does it indicate the “nn.Transformer” in PyTorch native or is there any other “Transformer” class in the accelerate? I am new to this library.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [March 8, 2023, 2:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-bloomforcausallm-from-sharded-checkpoints%2F32201%2F6 "2023-03-08T02:05:03Z")

</div>

Transformers the library you are using to load the model. `AutoModelForCausalLM.from_pretrained` accepts `device_map="sequential"`.

---

<div class="post-metadata">

**Author:** ![sh0416](https://avatars.discourse-cdn.com/v4/letter/s/dc4da7/32.png) [@sh0416](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsh0416)%5C
**Post date:** [March 8, 2023, 4:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-bloomforcausallm-from-sharded-checkpoints%2F32201%2F7 "2023-03-08T16:29:13Z")

</div>

It works using `AutoModelForCausalLM.from_pretrained("facebook/opt-6.7b", device_map="auto")`.

Thank you 🙂 It really helps.

Also, I think that the inferred device map doesn’t need no module split.

It already handles depending on the architecture. Is it right?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [March 8, 2023, 4:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-bloomforcausallm-from-sharded-checkpoints%2F32201%2F8 "2023-03-08T16:39:32Z")

</div>

Yes Transformers puts that info for you 🙂