# How to do classification fine-tuning of quantized models?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-do-classification-fine-tuning-of-quantized-models%2F71561%3E%5C
**Category:** 🤗Transformers\
**Created:** [February 2, 2024, 5:49am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-do-classification-fine-tuning-of-quantized-models%2F71561 "2024-02-02T05:49:37Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jstar2024](https://avatars.discourse-cdn.com/v4/letter/j/57b2e6/32.png) [@Jstar2024](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJstar2024)%5C
**Post date:** [February 2, 2024, 5:49am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-do-classification-fine-tuning-of-quantized-models%2F71561%2F1 "2024-02-02T05:49:37Z")

</div>

Classification fine-tuning of a non-quantized model works as expected, from the following code (assume all imports and unknown symbols are pre-defined) :

```auto
model = AutoModelForSequenceClassification.from_pretrained(
    "gpt2",
    num_labels=2,
    id2label=id2label, 
    label2id=label2id
)

lora_config = LoraConfig(
    task_type=TaskType.SEQ_CLS, 
    r=1,
    lora_alpha=8,
    lora_dropout=0.5,
)

model = get_peft_model(model, lora_config)
model.config.pad_token_id = model.config.eos_token_id

training_args = TrainingArguments(
    output_dir="test_trainer", 
    learning_rate=2e-5,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    eval_steps=50,
    logging_steps=50,
    evaluation_strategy="steps",
    num_train_epochs=3,
    weight_decay=0.01,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset['train'],
    eval_dataset=tokenized_dataset['eval'],
    compute_metrics=compute_metrics,
    data_collator=data_collator,
)

trainer.train()

```

With losses being logged as:

```auto
Step Training Loss Validation Loss Accuracy
50 0.699500 1.295200 0.514000
100 0.673200 1.224600 0.513400
...

```

But, when loading the model in 4bit with the following change (subsequent lines remaining the same) :

```auto
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForSequenceClassification.from_pretrained(
    "gpt2",
    quantization_config=bnb_config,
    num_labels=2,
    id2label=id2label, 
    label2id=label2id
)
... 

```

The training log is not as expected:

```auto
Step Training Loss Validation Loss Accuracy
50 3.699500 nan 1.000000
100 0.000000 nan 1.000000
...

```

And the model does not perform well after training.

Is there anything that I am missing or implementing incorrectly?