# Attention weights transfer but different classes

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fattention-weights-transfer-but-different-classes%2F32226%3E%5C
**Category:** Models\
**Created:** [February 21, 2023, 11:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fattention-weights-transfer-but-different-classes%2F32226 "2023-02-21T23:01:01Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![poix3](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/poix3/32/14307_2.png) [@poix3](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpoix3)%5C
**Post date:** [February 21, 2023, 11:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fattention-weights-transfer-but-different-classes%2F32226%2F1 "2023-02-21T23:01:01Z")

</div>

I want to apply the weights of attention layers of legal-bert (BertSelfAttention class) to longformer local attention QKV (LongformerSelfAttention class):

```auto
for layer_id in range(12):
    for new_layer, original_layer in zip(longformer.encoder.layer, legal_bert.encoder.layer):
        new_layer.attention.self.key.weight = 

```
