# How to fine tune BertForSequenceClassification with PEFT?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-fine-tune-bertforsequenceclassification-with-peft%2F39435%3E%5C
**Category:** Intermediate\
**Created:** [May 10, 2023, 7:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-fine-tune-bertforsequenceclassification-with-peft%2F39435 "2023-05-10T19:29:58Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![fredguth](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fredguth/32/2843_2.png) [@fredguth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffredguth)%5C
**Post date:** [May 10, 2023, 7:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-fine-tune-bertforsequenceclassification-with-peft%2F39435%2F1 "2023-05-10T19:29:58Z")

</div>

First I tried:

```auto
from transformers import AutoTokenizer, AutoConfig, AutoModelForSequenceClassification
from peft import (
    get_peft_config,
    get_peft_model,
    get_peft_model_state_dict,
    set_peft_model_state_dict,
    PeftType,
    PeftConfig,
)
checkpoint = "google/bert_uncased_L-4_H-256_A-4"

model = AutoModelForSequenceClassification.from_pretrained(
    checkpoint,
    load_in_8bit=True, 
    device_map='auto',
    num_labels=len(item), 
    )

```

But `BertForSequenceClassification` has no `device_map`… ok, I tried to use LORA without bitsandbytes.

But I also couldn’t do it:

```auto
peft_config = PeftConfig(PeftType.LORA, task_type="SEQ_CLS", base_model_name_or_path="BertForSequenceClassification")
model = AutoModelForSequenceClassification.from_pretrained(
    checkpoint,
    num_labels=len(item), 
    )
model = get_peft_model(model, peft_config)

```

Now the error is that `'PeftConfig' object has no attribute 'target_modules'`

Even the [collab in the PEFT blog post](https://huggingface.co/blog/peft) didn’t work for me because bitsandbytes expects a different CUDA version.

Is there a place with working examples?