# LongBlender embedding positions mismatch

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flongblender-embedding-positions-mismatch%2F5626%3E%5C
**Category:** 🤗Transformers\
**Created:** [April 19, 2021, 9:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flongblender-embedding-positions-mismatch%2F5626 "2021-04-19T21:07:07Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Alokin](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alokin/32/1532_2.png) [@Alokin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAlokin)%5C
**Post date:** [April 19, 2021, 9:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flongblender-embedding-positions-mismatch%2F5626%2F1 "2021-04-19T21:07:07Z")

</div>

I have successfully implemented Longformer’s self attention in BlenderBot Small. However, when I try to load the new model: BlenderbotSmallLongForConditionalGeneration .from\_pretrained(‘facebook/blenderbot\_small-4096’) , this error occurs:

RuntimeError: Error(s) in loading state\_dict for BlenderbotSmallLongForConditionalGeneration:  
size mismatch for model.encoder.embed\_positions.weight: copying a param with shape torch.Size([4096, 512]) from checkpoint, the shape in current model is torch.Size([512, 512]).

The mismatch between the embedding positions is intentional. I have used this project-  
[GitHub - patil-suraj/longbart: A long version of BART model based on Longformer model](https://github.com/patil-suraj/longbart) as an example and apparently, during transformers-2.10.0 this embedding positions mismatch was not a problem, because the author of the paper successfully loaded the Long version of Bart(which is similar to Blender). I have tried loading BartForConditionalGeneration.from\_pretrained as well, but the same problem occurred again.

My question is: How should I load the new model?

Thanks a lot 🤗