# Trainable weights in automodel and comparison with lora

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainable-weights-in-automodel-and-comparison-with-lora%2F37994%3E%5C
**Category:** 🤗Transformers\
**Created:** [April 28, 2023, 6:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainable-weights-in-automodel-and-comparison-with-lora%2F37994 "2023-04-28T06:56:02Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![prajwal56](https://avatars.discourse-cdn.com/v4/letter/p/bc79bd/32.png) [@prajwal56](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fprajwal56)%5C
**Post date:** [April 28, 2023, 6:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainable-weights-in-automodel-and-comparison-with-lora%2F37994%2F1 "2023-04-28T06:56:02Z")

</div>

Hi all ,

if i use RobertaForSequenceClassification or AutoModelForSequenceClassification,  
are all the weights trained or only the new classification head trained

also ,

i am also noticing for “roberta-large” ,peft lora is under performing AutoModelForSequenceClassification  
for glue data and benchmark.

peft lora lr=2e-4 , epoch =10,{‘accuracy’: 0.8897058823529411, ‘f1’: 0.9168207024029574}

AutoModelForSequenceClassification, lr=2e-5,epoch =10 {‘accuracy’: 0.9044117647058824, ‘f1’: 0.9312169312169313}

is something wrong , or is lora only useful for larger models