# Custom modification on transformers

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-modification-on-transformers%2F91930%3E%5C
**Category:** 🤗Transformers\
**Created:** [June 13, 2024, 8:42am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-modification-on-transformers%2F91930 "2024-06-13T08:42:34Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ronit33](https://avatars.discourse-cdn.com/v4/letter/r/e79b87/32.png) [@ronit33](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fronit33)%5C
**Post date:** [June 13, 2024, 8:42am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-modification-on-transformers%2F91930%2F1 "2024-06-13T08:42:34Z")

</div>

Are there any resources in huggingface on adding or removing layers in an already build model like bert or roberta.  
I want to replace some layers in a transformer model for my project. for example changing the self attention layer to some other kind of attention  
i also want to add my own layer inside the transformer as well.

Thank you.

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [June 13, 2024, 12:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-modification-on-transformers%2F91930%2F2 "2024-06-13T12:24:14Z")

</div>

Hi,

The Transformers library is not really aimed for this use case. It’s not meant to be a modular toolbox but rather aimed at people who want to use and fine-tune pre-trained models.

Of course, you could fork the Transformers library and tweak [modeling\_bert.py](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/modeling_bert.py) yourself, but if you want to hack around I’d recommend checking out other projects such as the ones from [Phil Wang](https://github.com/lucidrains) or [GitHub - facebookresearch/xformers: Hackable and optimized Transformers building blocks, supporting a composable construction.](https://github.com/facebookresearch/xformers).