# Enabling Flash Attention 2

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fenabling-flash-attention-2%2F87652%3E%5C
**Category:** 🤗Transformers\
**Created:** [May 21, 2024, 12:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fenabling-flash-attention-2%2F87652 "2024-05-21T12:50:04Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![varadhbhatnagar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/varadhbhatnagar/32/31383_2.png) [@varadhbhatnagar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvaradhbhatnagar)%5C
**Post date:** [May 21, 2024, 12:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fenabling-flash-attention-2%2F87652%2F1 "2024-05-21T12:50:04Z")

</div>

What is the difference between using Flash Attention 2 via

`model = AutoModelForCausalLM.from_pretrained(ckpt, attn_implementation = "sdpa")`

vs

`model = AutoModelForCausalLM.from_pretrained(ckpt, attn_implementation = "flash_attention_2")`

when Pytorch SDPA support FA2 according to [docs](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.htm) ?

@marcsun13

---

<div class="post-metadata">

**Author:** ![varadhbhatnagar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/varadhbhatnagar/32/31383_2.png) [@varadhbhatnagar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvaradhbhatnagar)%5C
**Post date:** [May 29, 2024, 9:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fenabling-flash-attention-2%2F87652%2F2 "2024-05-29T09:54:33Z")

</div>

@ybelkada Can you shed some light on this?

---

<div class="post-metadata">

**Author:** ![saireddy](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@saireddy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsaireddy)%5C
**Post date:** [July 3, 2024, 2:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fenabling-flash-attention-2%2F87652%2F3 "2024-07-03T14:41:43Z")

</div>

@varadhbhatnagar were you able to figure out the difference?