# Error while fine-tuning distilbert model **URL:** **Author:** ![AbdulrahmanAhmed](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/abdulrahmanahmed/32/32380_2.png) [@AbdulrahmanAhmed](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAbdulrahmanAhmed)%5C **Post date:** [November 17, 2024, 12:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferror-while-fine-tuning-distilbert-model%2F124588%2F1 "2024-11-17T00:24:06Z") ```auto from transformers import AutoModelForSequenceClassification from peft import LoraConfig, get_peft_model, TaskType import torch quantization_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype= torch.float16 ) model = AutoModelForSequenceClassification.from_pretrained( "distilbert-base-uncased" , quantization_config = quantization_config ) lora_config = LoraConfig( r=8, lora_alpha=8, target_modules=["q_lin", "k_lin", "v_lin", "out_lin"], lora_dropout=0.1, task_type=TaskType.SEQ_CLS, bias="none" ) model = get_peft_model(model, lora_config) ``` i tried to quantize and fine-tune distilbert-base-uncased using BnB and peft but I got this error message. ```auto low_cpu_mem_usage` was None, now default to True since model is quantized. Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) in () 24 ) 25 ---> 26 model = get_peft_model(model, lora_config) 5 frames /usr/local/lib/python3.10/dist-packages/peft/mapping.py in get_peft_model(model, peft_config, adapter_name, mixed, autocast_adapter_dtype, revision) 191 if peft_config.is_prompt_learning: 192 peft_config = _prepare_prompt_learning_config(peft_config, model_config) --> 193 return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type]( 194 model, peft_config, adapter_name=adapter_name, autocast_adapter_dtype=autocast_adapter_dtype 195 ) /usr/local/lib/python3.10/dist-packages/peft/peft_model.py in __init__ (self, model, peft_config, adapter_name, **kwargs) 1396 1397 # to make sure classifier layer is trainable; this may add a new ModulesToSaveWrapper -> 1398 _set_trainable(self, adapter_name) 1399 1400 def add_adapter(self, adapter_name: str, peft_config: PeftConfig) -> None: /usr/local/lib/python3.10/dist-packages/peft/utils/other.py in _set_trainable(model, adapter_name) 401 target.set_adapter(target.active_adapter) 402 else: --> 403 new_module = ModulesToSaveWrapper(target, adapter_name) 404 new_module.set_adapter(adapter_name) 405 setattr(parent, target_name, new_module) /usr/local/lib/python3.10/dist-packages/peft/utils/other.py in __init__ (self, module_to_save, adapter_name) 197 self._active_adapter = adapter_name 198 self._disable_adapters = False --> 199 self.update(adapter_name) 200 self.check_module() 201 /usr/local/lib/python3.10/dist-packages/peft/utils/other.py in update(self, adapter_name) 261 self.original_module.requires_grad_(False) 262 if adapter_name == self.active_adapter: --> 263 self.modules_to_save[adapter_name].requires_grad_(True) 264 265 def _create_new_hook(self, old_hook): /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in requires_grad_(self, requires_grad) 2885 """ 2886 for p in self.parameters(): -> 2887 p.requires_grad_(requires_grad) 2888 return self 2889 RuntimeError: only Tensors of floating point dtype can require gradients ``` it appears only when choosing task\_type = TaskType.SEQ\_CLS or TOKEN\_CLS inside LoraConfig. if I choose any other type like QUESTION\_ANS or CAUSAL\_LM no error occurs. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C **Post date:** [November 17, 2024, 2:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ferror-while-fine-tuning-distilbert-model%2F124588%2F2 "2024-11-17T02:17:57Z")
Similar to this case. > > > \### System Info > > > Python 3.11.5 > torch 2.3.0 > tran…sformers 4.41.1 > accelerate 0.30.1 > > \`\`\` > +---------------------------------------------------------------------------------------+ > | NVIDIA-SMI 545.23.06 Driver Version: 545.23.06 CUDA Version: 12.3 | > |-----------------------------------------+----------------------+----------------------+ > | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | > | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | > | | | MIG M. | > |=========================================+======================+======================| > | 0 NVIDIA A100-SXM4-80GB Off | 00000000:D6:00.0 Off | 0 | > | N/A 34C P0 61W / 400W | 4MiB / 81920MiB | 0% Default | > | | | Disabled | > +-----------------------------------------+----------------------+----------------------+ > | 1 NVIDIA A100-SXM4-80GB Off | 00000000:DA:00.0 Off | 0 | > | N/A 36C P0 60W / 400W | 4MiB / 81920MiB | 0% Default | > | | | Disabled | > +-----------------------------------------+----------------------+----------------------+ > > +---------------------------------------------------------------------------------------+ > | Processes: | > | GPU GI CI PID Type Process name GPU Memory | > | ID ID Usage | > |=======================================================================================| > | No running processes found | > +---------------------------------------------------------------------------------------+ > \`\`\` > > \### Reproduction > > > I want to fine tune Llama 3 70B with HF TRL. I am trying Accelerate, bitsandbytes quantization, mixed precision, FSDP on two GPUs with 80 GBs each. > > Running this code > \`\`\` > if rank == 0: > hf\_model = LlamaForCausalLM.from\_pretrained( > "meta-llama/Meta-Llama-3-70B-Instruct", > load\_in\_8bit=True, > device\_map="auto", > ) > \`\`\` > > via this command: > > \`\`\` > \> accelerate launch --config\_file ./accelerate\_fsdp\_config.yaml train.py > \`\`\` > > At Slurm managed cluster inside sbatch script with: > \`\`\` > \#SBATCH --nodes=1 > \#SBATCH --gpus-per-node=2 > \`\`\` > > with this fsdp config: > \`\`\` > compute\_environment: LOCAL\_MACHINE > debug: false > distributed\_type: FSDP > downcast\_bf16: 'no' > enable\_cpu\_affinity: false > fsdp\_config: > fsdp\_auto\_wrap\_policy: TRANSFORMER\_BASED\_WRAP > fsdp\_backward\_prefetch: BACKWARD\_PRE > fsdp\_cpu\_ram\_efficient\_loading: true > fsdp\_forward\_prefetch: false > fsdp\_offload\_params: true > fsdp\_sharding\_strategy: FULL\_SHARD > fsdp\_state\_dict\_type: SHARDED\_STATE\_DICT > fsdp\_sync\_module\_states: true > fsdp\_use\_orig\_params: true > machine\_rank: 0 > main\_training\_function: main > mixed\_precision: bf16 > num\_machines: 1 > num\_processes: 2 > rdzv\_backend: static > same\_network: true > tpu\_env: \[\] > tpu\_use\_cluster: false > tpu\_use\_sudo: false > use\_cpu: false > \`\`\` > > Results in: > \> \[rank1\]: RuntimeError: Only Tensors of floating point and complex dtype can require gradients > > the same for rank 0. The same when I do \`load\_in\_8bit=True\`. > > > > Error: > \`\`\` > \[rank0\]: Traceback (most recent call last): > \[rank0\]: File "/data/artyom\_karpov/rl4steg/train.py", line 345, in \ > \[rank0\]: train(context) > \[rank0\]: File "/data/artyom\_karpov/rl4steg/train.py", line 83, in train > \[rank0\]: hf\_model = LlamaForCausalLM.from\_pretrained( > \[rank0\]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > \[rank0\]: File "/data/artyom\_karpov/rl4steg/.venv/lib/python3.11/site-packages/transformers/modeling\_utils.py", line 3754, in from\_pretrained > \[rank0\]: ) = cls.\_load\_pretrained\_model( > \[rank0\]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^ > \[rank0\]: File "/data/artyom\_karpov/rl4steg/.venv/lib/python3.11/site-packages/transformers/modeling\_utils.py", line 4214, in \_load\_pretrained\_model > \[rank0\]: new\_error\_msgs, offload\_index, state\_dict\_index = \_load\_state\_dict\_into\_meta\_model( > \[rank0\]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > \[rank0\]: File "/data/artyom\_karpov/rl4steg/.venv/lib/python3.11/site-packages/transformers/modeling\_utils.py", line 896, in \_load\_state\_dict\_into\_meta\_model > \[rank0\]: value = type(value)(value.data.to("cpu"), \*\*value.\_\_dict\_\_) > \[rank0\]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > \[rank0\]: File "/data/artyom\_karpov/rl4steg/.venv/lib/python3.11/site-packages/bitsandbytes/nn/modules.py", line 297, in \_\_new\_\_ > \[rank0\]: return torch.Tensor.\_make\_subclass(cls, data, requires\_grad) > \[rank0\]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > \[rank0\]: RuntimeError: Only Tensors of floating point and complex dtype can require gradients > > \`\`\` > > > \### Expected behavior > > > I expect the model to be loaded.