# Understanding what went wrong in attention
**URL:**
**Author:**  [@prajjwal1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fprajjwal1)%5C
**Post date:** [July 20, 2020, 5:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funderstanding-what-went-wrong-in-attention%2F386%2F1 "2020-07-20T05:27:08Z")
I am working on attention analysis. I want to learn more about where self attention made mistakes while attending to context query. Given two sentences, I am interested in learning more about where self-attention should have paid more attention (and not irrelevant tokens) to provide correct answers. In general, what went wrong in processing a given sample even if fine-tuned transformer is employed.
While there are projects based on visualization like `BertViz`, `ExBERT`, I am not sure if it’s straightforward to extract the information I’m looking for.
Do you know of any good projects, or workarounds in `Transformers` to answer my query ?
---
**Author:**  [@prajjwal1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fprajjwal1)%5C
**Post date:** [July 21, 2020, 3:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funderstanding-what-went-wrong-in-attention%2F386%2F2 "2020-07-21T03:30:49Z")
Can anyone point me to the method on how to visualize attention in matrix form between query and context sentences ? Is there any other alternative ? Any pointers will be appreciated.
---
**Author:**  [@joeddav](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjoeddav)%5C
**Post date:** [July 24, 2020, 6:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funderstanding-what-went-wrong-in-attention%2F386%2F3 "2020-07-24T18:48:54Z")
The two you mentioned are the only I know of off the top of my head in terms of visualization tools. What are you trying to do that `BertViz` and `ExBERT` don’t provide? (disclaimer: not an expert in this area)
One tricky thing is that the notion of where the model _should_ or _should not_ have paid more attention is not well defined. There’s been debate about whether attention weights can or should be used for interpretation, for example see [[1]](https://arxiv.org/abs/1902.10186), [[2]](https://arxiv.org/abs/1908.04626). Coming up with a convincing argument that a given attention matrix should be one way or the other would probably not be trivial.
---
**Author:**  [@prajjwal1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fprajjwal1)%5C
**Post date:** [July 25, 2020, 4:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funderstanding-what-went-wrong-in-attention%2F386%2F4 "2020-07-25T04:52:15Z")
Thanks for your helpful reply. I had a look at their abstracts, and do not have a firm opinion on whether attention can fully help us understand what I’m looking for.
Both are good tools for interactive visualization but I want something that provide some quantifiable-ness. For now, I’m using `srush` way of visualizing attention heatmaps like he did in `Annotated Transformer`. Since I need to report in the paper, I am looking for static visualizations.
Based on my little interaction with `exbert` live demo, it can be hard for the reader to distinguish between what both models are looking at (for comparsion purposes).
For my use-case, I want the reader to be able to distinguish what two networks look at and how one is better than other. I hope it makes some sense.
---
**Author:**  [@prajjwal1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fprajjwal1)%5C
**Post date:** [July 29, 2020, 7:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funderstanding-what-went-wrong-in-attention%2F386%2F5 "2020-07-29T07:34:23Z")
@joeddav Could you please suggest what’s the recommended way to do what Exbert does with our own weights (seeing which token in sentence the model pays attention to) ? HF Exbert works for default pretrained LMs, I want my trained weights to be used for inference task. I’m running experiments on server, building npm and other stuff seems like a lot of work, but I think things may have changed a bit after introduction on HF inference API. I’m using `bert-base-uncased` (pretty standard), want to load weights from HF model hub instead.
---
**Author:**  [@prajjwal1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fprajjwal1)%5C
**Post date:** [July 31, 2020, 2:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funderstanding-what-went-wrong-in-attention%2F386%2F6 "2020-07-31T14:19:04Z")
Got it working by using exBERT locally.