# Swapping GPT-2 Attention with Flash Attention

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fswapping-gpt-2-attention-with-flash-attention%2F30244%3E%5C
**Category:** 🤗Transformers\
**Created:** [January 23, 2023, 8:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fswapping-gpt-2-attention-with-flash-attention%2F30244 "2023-01-23T20:57:49Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![conceptofmind](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/conceptofmind/32/55565_2.png) [@conceptofmind](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fconceptofmind)%5C
**Post date:** [January 23, 2023, 8:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fswapping-gpt-2-attention-with-flash-attention%2F30244%2F1 "2023-01-23T20:57:49Z")

</div>

Hi all,

Is there currently a way to extract the attention attribute from a model such as GPT-2 and swap it with Flash-Attention?

Thank you,

Enrico

---

<div class="post-metadata">

**Author:** ![iliemihai](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/iliemihai/32/14801_2.png) [@iliemihai](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Filiemihai)%5C
**Post date:** [March 18, 2023, 10:23am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fswapping-gpt-2-attention-with-flash-attention%2F30244%2F2 "2023-03-18T10:23:19Z")

</div>

I think you can multiplicate the positional embeddings, from what I have read, but it s not empirically tested.

---

<div class="post-metadata">

**Author:** ![conceptofmind](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/conceptofmind/32/55565_2.png) [@conceptofmind](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fconceptofmind)%5C
**Post date:** [March 18, 2023, 3:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fswapping-gpt-2-attention-with-flash-attention%2F30244%2F3 "2023-03-18T15:22:16Z")

</div>

I forgot to close this out. Resolved it awhile ago. You can swap the attention layers by building a wrapper.

---

<div class="post-metadata">

**Author:** ![AnanthZeke](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ananthzeke/32/48678_2.png) [@AnanthZeke](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAnanthZeke)%5C
**Post date:** [June 4, 2023, 3:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fswapping-gpt-2-attention-with-flash-attention%2F30244%2F4 "2023-06-04T15:25:45Z")

</div>

Can you share your code on how to swap the standard attention with flash attention on HF models?