# Unable to load 8bit model in Kaggle with dual GPU **URL:** **Author:** ![rcshubhadeep](https://avatars.discourse-cdn.com/v4/letter/r/c57346/32.png) [@rcshubhadeep](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frcshubhadeep)%5C **Post date:** [March 26, 2023, 4:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-load-8bit-model-in-kaggle-with-dual-gpu%2F34682%2F1 "2023-03-26T04:48:21Z") Hello All, I am trying to load meta/opt6.7B in Kaggle with dual T4 GPU. The code is simple and I could run that in colab. But I can’t make it work in Kaggle. What am I missing? Here is the code - ```auto !pip install bitsandbytes datasets accelerate loralib !pip install transformers peft import os # os.environ["CUDA_VISIBLE_DEVICES"]="0" import torch import torch.nn as nn import bitsandbytes as bnb from transformers import AutoTokenizer, AutoConfig, AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained( "facebook/opt-6.7b", load_in_8bit=True, device_map='auto', ) tokenizer = AutoTokenizer.from_pretrained("facebook/opt-6.7b") ``` And here is the stack trace. ```auto --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /tmp/ipykernel_23/3843057406.py in 9 "facebook/opt-6.7b", 10 load_in_8bit=True, ---> 11 device_map='auto', 12 ) 13 /opt/conda/lib/python3.7/site-packages/transformers/models/auto/auto_factory.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs) 463 model_class = _get_model_class(config, cls._model_mapping) 464 return model_class.from_pretrained( --> 465 pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs,** kwargs 466 ) 467 raise ValueError( /opt/conda/lib/python3.7/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs) 2527 # Dispatch model with hooks on all devices if necessary 2528 if device_map is not None: -> 2529 dispatch_model(model, device_map=device_map, offload_dir=offload_folder, offload_index=offload_index) 2530 2531 if output_loading_info: TypeError: dispatch_model() got an unexpected keyword argument 'offload_index' ``` Any idea what can be done? I am a bit clueless. Any help is much appreciated. ---
**Author:** ![rcshubhadeep](https://avatars.discourse-cdn.com/v4/letter/r/c57346/32.png) [@rcshubhadeep](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frcshubhadeep)%5C **Post date:** [March 26, 2023, 5:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-load-8bit-model-in-kaggle-with-dual-gpu%2F34682%2F2 "2023-03-26T05:05:16Z")
Ok this was super easy actually. I needed to upgrade the packages. Kaggle comes with pre-installed versions and they were old. So that is that. I have solved it 🙂 ---
**Author:** ![AayushShah](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aayushshah/32/22671_2.png) [@AayushShah](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAayushShah)%5C **Post date:** [April 3, 2023, 10:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-load-8bit-model-in-kaggle-with-dual-gpu%2F34682%2F3 "2023-04-03T10:48:04Z")
@rcshubhadeep Hello, can you share which packages you upgraded, please? I am receiving the same issue!! I have tried re-installing the transformers, but no luck. ---
**Author:** ![rcshubhadeep](https://avatars.discourse-cdn.com/v4/letter/r/c57346/32.png) [@rcshubhadeep](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frcshubhadeep)%5C **Post date:** [April 3, 2023, 10:51am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-load-8bit-model-in-kaggle-with-dual-gpu%2F34682%2F4 "2023-04-03T10:51:25Z")
I forgot to update this thread but I could not make it work there despite many attempts, unfortunately. I will try later. To answer your question, I updated transformers, datasets etc. Kaggle run time comes with a older version that makes this error. ---
**Author:** ![AayushShah](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aayushshah/32/22671_2.png) [@AayushShah](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAayushShah)%5C **Post date:** [April 3, 2023, 11:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-load-8bit-model-in-kaggle-with-dual-gpu%2F34682%2F5 "2023-04-03T11:04:18Z")
I have made it work… Please try the following. ```sh !pip uninstall wandb --yes !pip install --upgrade git+https://github.com/huggingface/transformers.git@main !pip install --upgrade bitsandbytes datasets accelerate loralib !pip install --upgrade git+https://github.com/huggingface/peft.git ``` The key is to **upgrade** the packages. So, I did that and that worked for me. Hope this helps. ---
**Author:** ![rcshubhadeep](https://avatars.discourse-cdn.com/v4/letter/r/c57346/32.png) [@rcshubhadeep](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frcshubhadeep)%5C **Post date:** [April 3, 2023, 11:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-load-8bit-model-in-kaggle-with-dual-gpu%2F34682%2F6 "2023-04-03T11:15:11Z")
Thanks for the update. Will check it out!