# Trying to understand XForSequenceClassification heads

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrying-to-understand-xforsequenceclassification-heads%2F1153%3E%5C
**Category:** Intermediate\
**Created:** [September 15, 2020, 11:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrying-to-understand-xforsequenceclassification-heads%2F1153 "2020-09-15T23:03:01Z")\
**Posts on this page:** 1\
**Showing post:** 5

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBramVanroy)%5C
**Post date:** [September 23, 2020, 2:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrying-to-understand-xforsequenceclassification-heads%2F1153%2F5 "2020-09-23T14:22:11Z")

</div>

When answering [this question](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fxlnetforsqeuenceclassification-warnings%2F1096%2F13)%2C I found that BertForSequenceClassification’s doesn’t actually use the pretrained linear weights for the classification layer. I kinda had expected that it did. It probably would not be useful for most tasks and needs finetuning anyway, but still.

Maybe the XForX models can have a line in their docs stating which of its heads are pretrained and which ones aren’t.

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrying-to-understand-xforsequenceclassification-heads%2F1153)._