# Issue with LlamaSdpaAttention Not Being Utilized
**URL:**
**Author:**  [@jacobvinje](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjacobvinje)%5C
**Post date:** [February 13, 2025, 12:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-with-llamasdpaattention-not-being-utilized%2F140678%2F1 "2025-02-13T12:16:37Z")
Hello everyone,
I’m working with the Llama model from Hugging Face Transformers (v4.48.3) and noticed that it’s using LlamaAttention instead of LlamaSdpaAttention by default. This seems unexpected since my understanding is that the model should automatically use the SDPA kernel (torch.nn.functional.scaled\_dot\_product\_attention) when possible.
Here’s my minimal reproduction:
```auto
from transformers import AutoTokenizer, AutoModelForCausalLM
model_name = "meta-llama/Llama-3.2-1B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
print(model) # Shows LlamaAttention, not LlamaSdpaAttention
```
Even though I’m not requesting attention outputs or triggering any known conditions that would force a fallback to eager mode, the model still uses LlamaAttention. My environment:
- Python 3.10.14
- PyTorch 2.5.1
- Transformers 4.48.3
- CUDA 12.4.1
What determines whether LlamaSdpaAttention is used by default? Is there something specific about this version of Transformers or my setup that’s preventing automatic SDPA usage?
Also, when I try to set it to use SDPA manually, it still uses the normal/old LlamaAttention.
config.\_attn\_implementation = “sdpa”
model = AutoModelForCausalLM.from\_pretrained(model\_name, config=config)
Thanks for any insights!
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [February 13, 2025, 2:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-with-llamasdpaattention-not-being-utilized%2F140678%2F2 "2025-02-13T14:01:58Z")
I’ve confirmed that the bug can be reproduced. Your usage and torch version should be correct…
From the code, it doesn’t look like there are any other suspicious branches.
> **[huggingface/transformers](https://github.com/huggingface/transformers/issues)**
>
> 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX. - huggingface/transformers
>
---
**Author:**  [@karlexmarin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkarlexmarin)%5C
**Post date:** [October 25, 2026, 1:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-with-llamasdpaattention-not-being-utilized%2F140678%2F3 "2026-10-25T13:31:56Z")
A version-specific clarification for @jacobvinje and future readers: in **Transformers 4.48.3** , seeing `LlamaAttention` in `print(model)` does **not** by itself mean SDPA is disabled.
In that version, `LlamaAttention.forward` selects its attention function internally using `self.config._attn_implementation`. The implementation is visible in the [v4.48.3 source](https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/llama/modeling_llama.py).
For an existing loaded model, this is a useful first check:
```python
print(model.config._attn_implementation)
```
That reports the configured implementation, not proof of the GPU kernel actually executed. In this version, requesting `output_attentions=True` with SDPA triggers an eager-attention fallback. A profiler is needed to verify the executed operations and any memory implications.
So the unchanged class name is consistent with SDPA being selected; it is not, on its own, a reproducible backend failure. This clarification does not establish the cause of the separate memory issue you reported.
For a fuller explanation, I also maintain a free [Transformer guide (English and Spanish)](https://karlesmarin.github.io/transformers-guide/). The [KV-cache chapter](https://karlesmarin.github.io/transformers-guide/en/chapters/20-kv-cache.html) explains the memory budget and why fitting in memory, retaining attention mass and preserving task quality are separate questions.