# SmolVLM 8bit Quantization Problem

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsmolvlm-8bit-quantization-problem%2F128287%3E%5C
**Category:** Models\
**Created:** [November 28, 2024, 2:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsmolvlm-8bit-quantization-problem%2F128287 "2024-11-28T14:35:50Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![uisikdag](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/uisikdag/32/43363_2.png) [@uisikdag](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fuisikdag)%5C
**Post date:** [November 28, 2024, 2:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsmolvlm-8bit-quantization-problem%2F128287%2F1 "2024-11-28T14:35:50Z")

</div>

I quantize the HuggingFaceTB/SmolVLM-Instruct using the code below.(i.e. at the model’s page)

from transformers import AutoModelForVision2Seq, BitsAndBytesConfig  
import torch

quantization\_config = BitsAndBytesConfig(load\_in\_8bit=True)  
model = AutoModelForVision2Seq.from\_pretrained(  
“HuggingFaceTB/SmolVLM-Instruct”,  
quantization\_config=quantization\_config,  
)

Local inference after quantization works fine. As far as I understand something is happening during serialization, but upload to HF works also fine ,but when I try to download from my repository

uisikdag/SmolVLM-Instruct-8bit

in huggingface, and run the quantized model,I get an error regarding unmatched tensor sizes.Help will be appreciated.

Edit:This issue does not occur in 4bit quantization with nf4.That works totally fine.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [November 28, 2024, 3:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsmolvlm-8bit-quantization-problem%2F128287%2F2 "2024-11-28T15:19:53Z")

</div>

It was also reproduced beautifully here. It seems to be an issue with bitsandbytes that has been unresolved for a long time. The fact that it occurred in an official HF model may be a chance for a solution.😅

> <https://github.com/bitsandbytes-foundation/bitsandbytes/issues/1097>
>
> \### System Info
> 
> Hi Team,
> 
> when i am running the above qlora code for owl-vit …model (google/owlvit-base-patch32) with below 4 bits bnbconfig , the fine tuning is taking place without any error.
> 
> bnb\_config = BitsAndBytesConfig(
> load\_in\_4bit=True,
> bnb\_4bit\_use\_double\_quant=True,
> bnb\_4bit\_quant\_type="nf4",
> bnb\_4bit\_compute\_dtype=torch.bfloat16
> )
> 
> once i change the config with below information
> 
> bnb\_config = BitsAndBytesConfig(
> load\_in\_8bit=True
> )
> 
> i receive the following error trace.
> 
> \#########################################################################################
> RuntimeError Traceback (most recent call last)
> Cell In\[25\], line 1
> \----\> 1 trainer.train()
> 
> File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1537, in Trainer.train(self, resume\_from\_checkpoint, trial, ignore\_keys\_for\_eval, \*\*kwargs)
> 1535 hf\_hub\_utils.enable\_progress\_bars()
> 1536 else:
> \-\> 1537 return inner\_training\_loop(
> 1538 args=args,
> 1539 resume\_from\_checkpoint=resume\_from\_checkpoint,
> 1540 trial=trial,
> 1541 ignore\_keys\_for\_eval=ignore\_keys\_for\_eval,
> 1542 )
> 
> File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1854, in Trainer.\_inner\_training\_loop(self, batch\_size, args, resume\_from\_checkpoint, trial, ignore\_keys\_for\_eval)
> 1851 self.control = self.callback\_handler.on\_step\_begin(args, self.state, self.control)
> 1853 with self.accelerator.accumulate(model):
> \-\> 1854 tr\_loss\_step = self.training\_step(model, inputs)
> 1856 if (
> 1857 args.logging\_nan\_inf\_filter
> 1858 and not is\_torch\_tpu\_available()
> 1859 and (torch.isnan(tr\_loss\_step) or torch.isinf(tr\_loss\_step))
> 1860 ):
> 1861 # if loss is nan or inf simply add the average of previous logged losses
> 1862 tr\_loss += tr\_loss / (1 + self.state.global\_step - self.\_globalstep\_last\_logged)
> 
> File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:2744, in Trainer.training\_step(self, model, inputs)
> 2742 scaled\_loss.backward()
> 2743 else:
> \-\> 2744 self.accelerator.backward(loss)
> 2746 return loss.detach() / self.args.gradient\_accumulation\_steps
> 
> File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/accelerate/accelerator.py:1907, in Accelerator.backward(self, loss, \*\*kwargs)
> 1905 return
> 1906 elif self.scaler is not None:
> \-\> 1907 self.scaler.scale(loss).backward(\*\*kwargs)
> 1908 else:
> 1909 loss.backward(\*\*kwargs)
> 
> File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/\_tensor.py:492, in Tensor.backward(self, gradient, retain\_graph, create\_graph, inputs)
> 482 if has\_torch\_function\_unary(self):
> 483 return handle\_torch\_function(
> 484 Tensor.backward,
> 485 (self,),
> (...)
> 490 inputs=inputs,
> 491 )
> \--\> 492 torch.autograd.backward(
> 493 self, gradient, retain\_graph, create\_graph, inputs=inputs
> 494 )
> 
> File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/init.py:251, in backward(tensors, grad\_tensors, retain\_graph, create\_graph, grad\_variables, inputs)
> 246 retain\_graph = create\_graph
> 248 # The reason we repeat the same comment below is that
> 249 # some Python versions print out the first line of a multi-line function
> 250 # calls in the traceback and some print out the last line
> \--\> 251 Variable.execution\_engine.run\_backward( # Calls into the C++ engine to run the backward pass
> 252 tensors,
> 253 grad\_tensors,
> 254 retain\_graph,
> 255 create\_graph,
> 256 inputs,
> 257 allow\_unreachable=True,
> 258 accumulate\_grad=True,
> 259 )
> 
> File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/function.py:288, in BackwardCFunction.apply(self, \*args)
> 282 raise RuntimeError(
> 283 "Implementing both 'backward' and 'vjp' for a custom "
> 284 "Function is not allowed. You should only implement one "
> 285 "of them."
> 286 )
> 287 user\_fn = vjp\_fn if vjp\_fn is not Function.vjp else backward\_fn
> \--\> 288 return user\_fn(self, \*args)
> 
> File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/bitsandbytes/autograd/functions.py:491, in MatMul8bitLt.backward(ctx, grad\_output)
> 485 print("state.CxB",state.CxB)
> 486 print("State ",state)
> 488 CB = (
> 489 undo\_layout(state.CxB, state.tile\_indices)
> 490 .to(ctx.dtype\_A)
> \--\> 491 .mul(state.SCB.unsqueeze(1).mul(1.0 / 127.0))
> 492 )
> 493 grad\_A = torch.matmul(grad\_output, CB).view(ctx.grad\_shape).to(ctx.dtype\_A)
> 494 else:
> 
> RuntimeError: The size of tensor a (32) must match the size of tensor b (4) at non-singleton dimension 0
> 
> \#######################################################################################
> need your help.
> 
> thanks.
> 
> \### Reproduction
> 
> Need your help to solve this problem
> 
> \### Expected behavior
> 
> it is working fine for 4 bits quantization and finetuning but same code, dataset with 8 bit it is not working. it gives runtime error ( RuntimeError: The size of tensor a (32) must match the size of tensor b (4) at non-singleton dimension 0)

Edit:  
It seems that this is a different problem. It’s worse than the one above.

Edit:  
Maybe this issue.

> <https://github.com/unslothai/unsloth/issues/638>
>
> I would like to finetune CodeLlama-13b in a memory efficient way.
> 
> I was able …to do it with CodeLlama-7b, but failing with 13b.
> 
> I can't load the model \`unsloth/codellama-13b-bnb-4bit\`:
> 
> \`\`\`python
> model, tokenizer = unsloth.FastLanguageModel.from\_pretrained('codellama/CodeLlama-13b-hf', load\_in\_4bit=True)
> \`\`\`
> 
> \> ValueError: Supplied state dict for model.layers.28.mlp.gate\_proj.weight does not contain \`bitsandbytes\_\_\*\` and possibly other \`quantized\_stats\` components.
> 
> 
> I tried to quantize it first, but that also failed
> 
> \`\`\`python
> model, tokenizer = unsloth.FastLanguageModel.from\_pretrained('codellama/CodeLlama-13b-hf', load\_in\_4bit=False)
> model.save\_pretrained\_gguf('./codellama-13b-bnb-4bit', tokenizer=tokenizer)
> \`\`\`
> 
> \> RuntimeError: The weights trying to be saved contained shared tensors \[{'model.layers.26.self\_attn.q\_proj.weight', 'model.layers.31.self\_attn.v\_proj.weight', 'model.layers.32.self\_attn.q\_proj.weight', 'model.layers.39.self\_attn.q\_proj.weight', 'model.layers.26.self\_attn.o\_proj.weight', 'model.layers.35.self\_attn.q\_proj.weight', 'model.layers.28.self\_attn.o\_proj.weight', 'model.layers.33.self\_attn.q\_proj.weight', 'model.layers.29.self\_attn.v\_proj.weight', 'model.layers.33.self\_attn.k\_proj.weight', 'model.layers.35.self\_attn.k\_proj.weight', 'model.layers.29.self\_attn.o\_proj.weight', 'model.layers.36.self\_attn.q\_proj.weight', 'model.layers.36.self\_attn.k\_proj.weight', 'model.layers.37.self\_attn.q\_proj.weight', 'model.layers.39.self\_attn.k\_proj.weight', 'model.layers.30.self\_attn.o\_proj.weight', 'model.layers.27.self\_attn.q\_proj.weight', 'model.layers.27.self\_attn.k\_proj.weight', 'model.layers.28.self\_attn.v\_proj.weight', 'model.layers.38.self\_attn.k\_proj.weight', 'model.layers.34.self\_attn.q\_proj.weight', 'model.layers.33.self\_attn.v\_proj.weight', 'model.layers.32.self\_attn.o\_proj.weight', 'model.layers.28.self\_attn.q\_proj.weight', 'model.layers.33.self\_attn.o\_proj.weight', 'model.layers.36.self\_attn.v\_proj.weight', 'model.layers.31.self\_attn.q\_proj.weight', 'model.layers.30.self\_attn.v\_proj.weight', 'model.layers.32.self\_attn.v\_proj.weight', 'model.layers.34.self\_attn.v\_proj.weight', 'model.layers.31.self\_attn.k\_proj.weight', 'model.layers.27.self\_attn.v\_proj.weight', 'model.layers.29.self\_attn.q\_proj.weight', 'model.layers.27.self\_attn.o\_proj.weight', 'model.layers.30.self\_attn.q\_proj.weight', 'model.layers.34.self\_attn.o\_proj.weight', 'model.layers.36.self\_attn.o\_proj.weight', 'model.layers.39.self\_attn.v\_proj.weight', 'model.layers.39.self\_attn.o\_proj.weight', 'model.layers.34.self\_attn.k\_proj.weight', 'model.layers.32.self\_attn.k\_proj.weight', 'model.layers.26.self\_attn.v\_proj.weight', 'model.layers.35.self\_attn.v\_proj.weight', 'model.layers.37.self\_attn.v\_proj.weight', 'model.layers.29.self\_attn.k\_proj.weight', 'model.layers.28.self\_attn.k\_proj.weight', 'model.layers.37.self\_attn.o\_proj.weight', 'model.layers.37.self\_attn.k\_proj.weight', 'model.layers.35.self\_attn.o\_proj.weight', 'model.layers.31.self\_attn.o\_proj.weight', 'model.layers.26.self\_attn.k\_proj.weight', 'model.layers.38.self\_attn.q\_proj.weight', 'model.layers.30.self\_attn.k\_proj.weight', 'model.layers.38.self\_attn.v\_proj.weight', 'model.layers.38.self\_attn.o\_proj.weight'}, {'model.layers.37.mlp.gate\_proj.weight', 'model.layers.31.mlp.down\_proj.weight', 'model.layers.30.mlp.up\_proj.weight', 'model.layers.33.mlp.up\_proj.weight', 'model.layers.35.mlp.gate\_proj.weight', 'model.layers.31.mlp.gate\_proj.weight', 'model.layers.26.mlp.gate\_proj.weight', 'model.layers.27.mlp.down\_proj.weight', 'model.layers.30.mlp.gate\_proj.weight', 'model.layers.35.mlp.down\_proj.weight', 'model.layers.26.mlp.up\_proj.weight', 'model.layers.38.mlp.gate\_proj.weight', 'model.layers.33.mlp.down\_proj.weight', 'model.layers.29.mlp.up\_proj.weight', 'model.layers.30.mlp.down\_proj.weight', 'model.layers.36.mlp.down\_proj.weight', 'model.layers.29.mlp.down\_proj.weight', 'model.layers.33.mlp.gate\_proj.weight', 'model.layers.37.mlp.up\_proj.weight', 'model.layers.31.mlp.up\_proj.weight', 'model.layers.37.mlp.down\_proj.weight', 'model.layers.32.mlp.gate\_proj.weight', 'model.layers.39.mlp.down\_proj.weight', 'model.layers.34.mlp.down\_proj.weight', 'model.layers.39.mlp.gate\_proj.weight', 'model.layers.32.mlp.up\_proj.weight', 'model.layers.26.mlp.down\_proj.weight', 'model.layers.36.mlp.up\_proj.weight', 'model.layers.27.mlp.gate\_proj.weight', 'model.layers.34.mlp.gate\_proj.weight', 'model.layers.38.mlp.up\_proj.weight', 'model.layers.27.mlp.up\_proj.weight', 'model.layers.36.mlp.gate\_proj.weight', 'model.layers.38.mlp.down\_proj.weight', 'model.layers.35.mlp.up\_proj.weight', 'model.layers.28.mlp.up\_proj.weight', 'model.layers.28.mlp.down\_proj.weight', 'model.layers.32.mlp.down\_proj.weight', 'model.layers.28.mlp.gate\_proj.weight', 'model.layers.34.mlp.up\_proj.weight', 'model.layers.29.mlp.gate\_proj.weight', 'model.layers.39.mlp.up\_proj.weight'}, {'model.layers.37.input\_layernorm.weight', 'model.layers.32.post\_attention\_layernorm.weight', 'model.layers.35.input\_layernorm.weight', 'model.layers.35.post\_attention\_layernorm.weight', 'model.layers.31.input\_layernorm.weight', 'model.layers.26.input\_layernorm.weight', 'model.layers.36.input\_layernorm.weight', 'model.layers.34.post\_attention\_layernorm.weight', 'model.layers.27.post\_attention\_layernorm.weight', 'model.layers.27.input\_layernorm.weight', 'model.layers.37.post\_attention\_layernorm.weight', 'model.norm.weight', 'model.layers.28.post\_attention\_layernorm.weight', 'model.layers.38.post\_attention\_layernorm.weight', 'model.layers.34.input\_layernorm.weight', 'model.layers.30.input\_layernorm.weight', 'model.layers.38.input\_layernorm.weight', 'model.layers.30.post\_attention\_layernorm.weight', 'model.layers.29.post\_attention\_layernorm.weight', 'model.layers.32.input\_layernorm.weight', 'model.layers.28.input\_layernorm.weight', 'model.layers.31.post\_attention\_layernorm.weight', 'model.layers.39.input\_layernorm.weight', 'model.layers.33.input\_layernorm.weight', 'model.layers.26.post\_attention\_layernorm.weight', 'model.layers.39.post\_attention\_layernorm.weight', 'model.layers.29.input\_layernorm.weight', 'model.layers.36.post\_attention\_layernorm.weight', 'model.layers.33.post\_attention\_layernorm.weight'}\] that are mismatching the transformers base configuration. Try saving using \`safe\_serialization=False\` or remove this tensor sharing.
> 
> Is CodeLlama-13b not supported? Should I be using a different model?

> <https://stackoverflow.com/questions/79068298/valueerror-supplied-state-dict-for-layers-does-not-contain-bitsandbytes-an>

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [November 28, 2024, 5:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsmolvlm-8bit-quantization-problem%2F128287%2F3 "2024-11-28T17:37:53Z")

</div>

I’ve tried going back as far as possible in the library version, but I can’t avoid the error. I hope I’m just making an easy mistake…😪

```py
from transformers import AutoProcessor, AutoModelForVision2Seq, BitsAndBytesConfig
import torch
quantization_config = BitsAndBytesConfig(load_in_8bit=True)
#quantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)

temp_model_dir = "temp_model"
model_id = "HuggingFaceTB/SmolVLM-Instruct"
#model_id = "uisikdag/SmolVLM-Instruct-8bit" # bnb quantized
#model_id = "Salesforce/blip2-opt-2.7b" # for reproduction in older version transformers
temp_model = AutoModelForVision2Seq.from_pretrained(model_id, torch_dtype=torch.bfloat16, quantization_config=quantization_config)
temp_model.save_pretrained(temp_model_dir)
#temp_model_dir = model_id # if this is enabled, it will not crash
model = AutoModelForVision2Seq.from_pretrained(temp_model_dir, quantization_config=quantization_config) # crashes here
#processor = AutoProcessor.from_pretrained(temp_model_dir)

# dependencies
"""
torch==2.4.0
accelerate==1.0.0
huggingface_hub==0.26.0
transformers==4.46.0
bitsandbytes==0.44.0
numpy<2
peft==0.12.0
safetensors==0.4.3
"""

```

---

<div class="post-metadata">

**Author:** ![uisikdag](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/uisikdag/32/43363_2.png) [@uisikdag](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fuisikdag)%5C
**Post date:** [November 29, 2024, 12:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsmolvlm-8bit-quantization-problem%2F128287%2F4 "2024-11-29T00:25:24Z")

</div>

thx for trying !