# Using Seq2SeqTrainer for decoders?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-seq2seqtrainer-for-decoders%2F132982%3E%5C
**Category:** 🤗Transformers\
**Created:** [December 25, 2024, 8:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-seq2seqtrainer-for-decoders%2F132982 "2024-12-25T08:10:37Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![guymorlan](https://avatars.discourse-cdn.com/v4/letter/g/3da27b/32.png) [@guymorlan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fguymorlan)%5C
**Post date:** [December 25, 2024, 8:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-seq2seqtrainer-for-decoders%2F132982%2F1 "2024-12-25T08:10:37Z")

</div>

I have a case where I can’t use SFTTrainer because:

1. My data is pretokenized
2. I need to use predict\_with\_generate to get validation set evals during training

The regular Trainer can handle pretokenized data but doesn’t have predict\_with\_generate.  
Can I use Seq2SeqTrainer or does that only make sense for encoder-decoder models?

I’m fine tuning a 7B LLM with Qlora.

Thanks!