# Cache T5 encoder results within batch when training

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcache-t5-encoder-results-within-batch-when-training%2F4200%3E%5C
**Category:** 🤗Transformers\
**Created:** [March 6, 2021, 9:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcache-t5-encoder-results-within-batch-when-training%2F4200 "2021-03-06T09:28:38Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![marton-avrios](https://avatars.discourse-cdn.com/v4/letter/m/ecae2f/32.png) [@marton-avrios](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmarton-avrios)%5C
**Post date:** [March 6, 2021, 9:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcache-t5-encoder-results-within-batch-when-training%2F4200%2F1 "2021-03-06T09:28:38Z")

</div>

My batches are a little unorthodox, they consist of the same input but different targets. Basically I moved the prefixes T5 uses from the input to the target and trained the model this way. Like this:

input: “I am Alex, 33 years old”, target: “name: Alex”  
input: “I am Alex, 33 years old”, target: “age: 33”

It would make training much faster if I could calculate the encoder output just once per batch and reuse it within the batch. So 1 forward pass for the encoder but multiple backward passes. Is there a library supported way to do this or do I need to go completely custom and write the training loop from scratch?