# PEFT LORA for Text Classification?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpeft-lora-for-text-classification%2F42321%3E%5C
**Category:** 🤗Transformers\
**Created:** [June 6, 2023, 9:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpeft-lora-for-text-classification%2F42321 "2023-06-06T21:45:11Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![maadnfritz](https://avatars.discourse-cdn.com/v4/letter/m/9dc877/32.png) [@maadnfritz](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmaadnfritz)%5C
**Post date:** [June 6, 2023, 9:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpeft-lora-for-text-classification%2F42321%2F1 "2023-06-06T21:45:11Z")

</div>

I am working on a project for text classification. I started out with BERT and AutoModelForSequenceClassification and now i want to move up the food chain and try some larger models. to do that on a single A100 i was hoping to use PEFT, LORA and bitsandbytes or accelerate for example. Bit the examples i have found all use AutoModelForCausalLM.

I tried to adapt one of the tutorials to use BloomForSequenceClassification but the  
PEFT tutorial suggested " Finally, we need to apply some post-processing on the 8-bit model to enable training, let’s freeze all our layers, and cast the layer-norm in `float32` for stability. We also cast the output of the last layer in `float32` for the same reasons." and had code  
model.lm\_head = CastOutputToFloat(model.lm\_head)  
Which is fine for AutoModelForCausalLM but does not work for BloomForSequenceClassification because it does not have an lm\_head layer.

Which leads me to ask, can one use PEFT and LORA with a AutoModelForSequenceClassification?

Alternatively, can one use a AutoModelForCausalLM for text classification?

cheers

---

<div class="post-metadata">

**Author:** ![jonathanlyj](https://avatars.discourse-cdn.com/v4/letter/j/a9a28c/32.png) [@jonathanlyj](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjonathanlyj)%5C
**Post date:** [December 8, 2023, 6:12am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpeft-lora-for-text-classification%2F42321%2F2 "2023-12-08T06:12:24Z")

</div>

I have exactly the same problem. If removing CastOutputToFloat line entirely, another error is triggered. Is there a solution for this?