# Decoder generate with prompts of variable lengths?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecoder-generate-with-prompts-of-variable-lengths%2F18364%3E%5C
**Category:** 🤗Transformers\
**Created:** [May 25, 2022, 1:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecoder-generate-with-prompts-of-variable-lengths%2F18364 "2022-05-25T13:54:04Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sandstone5911](https://avatars.discourse-cdn.com/v4/letter/s/f05b48/32.png) [@Sandstone5911](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSandstone5911)%5C
**Post date:** [May 25, 2022, 1:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecoder-generate-with-prompts-of-variable-lengths%2F18364%2F1 "2022-05-25T13:54:04Z")

</div>

Hi,

Beginner here. I would like to know how I can batch-generate with a decoder where the input\_ids are initialized with prompts of **variable lengths**?

For example:  
I tokenized the prompts with _padding=True_, but the _generate( )_ function does not seem to consider the _attention\_mask_ provided. I have tried using beam search and sample and neither has worked.

```auto
text_decoder = BertLMHeadModel(config=my_config) # decoder
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

prompts = custom_prompts # list of strings with variable lengths
tokenized = self.tokenizer(prompt, padding=True, return_tensors="pt")
input_ids = tokenized.input_ids
att_msk = tokenized.attention_mask

outputs = text_decoder.generate(
            input_ids=input_ids,
            attention_mask = att_msk,
            max_length=30,
            min_length=5,
            do_sample=True,
            encoder_hidden_states=image_embeds, # size(0) == len(prompts)
            encoder_attention_mask=image_atts, # size(0) == len(prompts)
            top_p=0.9,
            num_return_sequences=1,
            eos_token_id=self.tokenizer.sep_token_id,
            pad_token_id=self.tokenizer.pad_token_id, 
            repetition_penalty=1.1,                                            
            **model_kwargs)

```

As shown in the outputs below, the generated sequences are appended after the padded tokens, while I need the decoder to only look at the sequence before the padding when predicting outputs.

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/3/382b967dd9289c1654344616bb8f4007f99b3dad.png)  
Can someone please give me some advice on how to achieve this or is there already a function to do this? Thank you very much!