# Quantization GPTQ

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquantization-gptq%2F85824%3E%5C
**Category:** 🤗Optimum\
**Created:** [May 8, 2024, 1:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquantization-gptq%2F85824 "2024-05-08T13:58:32Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![KashyapR](https://avatars.discourse-cdn.com/v4/letter/k/a87d85/32.png) [@KashyapR](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKashyapR)%5C
**Post date:** [May 8, 2024, 1:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquantization-gptq%2F85824%2F1 "2024-05-08T13:58:32Z")

</div>

Question: Quantization through GPTQ

Hi Team, I’m trying to quantize a 13b model using the below configuration on A100. I tried the below options

quantization\_config = GPTQConfig(  
bits=4,  
group\_size=128,  
dataset=“wikitext2”,  
batch\_size=16,  
desc\_act=False

)

1. Enforce batch\_size = 16 or batch\_size = 2 at the quant configurations
2. Set tokenizer.pad\_token\_id = tokenizer.eos\_token\_id (which is 2)

I observed that even if we explicitly enforce the batch size and set the pad\_token\_id value other than None. It is not being considered

Can’t we set the batch\_size and pad\_token\_id to some other value is this expected behavior with GPTQ . What is the reason behind this? Please suggest if there is any way to override the batch size config.

> <https://github.com/huggingface/optimum/blob/main/optimum/gptq/data.py#L51>

 ![Screenshot 2024-05-08 at 8.55.39 AM](https://us1.discourse-cdn.com/hellohellohello/original/3X/1/1/1176d9140fe9b413344b6c451f79cb24ef791c06.jpeg)

Could you kindly suggest? Appreciate your kind support.  
Thanks

---

<div class="post-metadata">

**Author:** ![IlyasMoutawwakil](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ilyasmoutawwakil/32/27200_2.png) [@IlyasMoutawwakil](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FIlyasMoutawwakil)%5C
**Post date:** [May 21, 2024, 12:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquantization-gptq%2F85824%2F2 "2024-05-21T12:56:12Z")

</div>

Hi! could you try passing the `pad_token_id` in the `GPTQConfig` quantization config, from reading the code it seems this is the value that’s been used in the dataset preparation.