# Dynamic attention mask during GPT-2 training

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdynamic-attention-mask-during-gpt-2-training%2F2789%3E%5C
**Category:** 🤗Transformers\
**Created:** [December 11, 2020, 12:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdynamic-attention-mask-during-gpt-2-training%2F2789 "2020-12-11T00:58:16Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![xwwwww](https://avatars.discourse-cdn.com/v4/letter/x/839c29/32.png) [@xwwwww](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fxwwwww)%5C
**Post date:** [December 11, 2020, 12:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdynamic-attention-mask-during-gpt-2-training%2F2789%2F1 "2020-12-11T00:58:16Z")

</div>

My task is to generate a list of options and a story, given an intro.

> an option is a sentence starting with a special token ‘\<|option|\>’.  
> an instance is something like “intro \<|endofintro|\> \<|option|\> option1 \<|option|\> option2 \<|endofoption\> story \< endofstory\>”

I need a dynamic attention mask because  
(1) As I don’t want the following options to attention to the previous options, I should mask all the previous options. For example, I could set the attention mask on all the options pos to 0.  
(2) However, as the story should attention to all the options, I should keep the attention mask on all the options pos to 1

I could simply implement that during generation by generating one option per time, and after generating options, I could modify the attention mask to generate the story.

But I don’t know how to do that during training.