# T5 Model Generate and Model Outputs Vastly Different
**URL:**
**Author:**  [@hockeybro12](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhockeybro12)%5C
**Post date:** [August 17, 2022, 1:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-model-generate-and-model-outputs-vastly-different%2F21715%2F1 "2022-08-17T01:46:30Z")
Hello,
I have finetuned a T5 model for generation, and it learns well and the loss decreases a lot. When I evaluate the model, I’m happy with the results when I do `model.forward`, but when I call `model.generate`, no matter what parameters I set for decoding, the model performs poorly. In fact, once the model has overfitted, `model.generate` something pretty similar every time. I’m unsure of what I’m doing wrong, and I’ve looked at lots of topics. It seems like right now I don’t even have to shift my inputs right for the decoder as long as I’m not passing decoder inputs, so I’m very unsure what the problem is.
Also, how do people normally decide what decoding method to use? Is it just based on what “looks good” on the validation set?
Here is my tokenizing code:
```auto
encoding = tokenizer(
[task_prefix + sequence for sequence in input_sequences],
#padding="longest",
max_length=max_source_length,
truncation=True,
return_tensors="pt",
padding="max_length",
)
target_tensor_input, t5_masks = encoding.input_ids, encoding.attention_mask
target_encoding = tokenizer(
output_sequences, padding="longest", max_length=max_target_length, truncation=True
)
labels = target_encoding.input_ids
# replace padding token id's of the labels by -100 so it's ignored by the loss
t5_outputs = torch.tensor(labels)
t5_outputs [t5_outputs == tokenizer.pad_token_id] = -100
```
Here is a minimalist version of my model training code:
```auto
for target_tensor_input, t5_outputs, t5_masks in tqdm(train_dataloader):
loss = model(input_ids=target_tensor_input.cuda(), attention_mask=t5_masks.cuda(), labels=t5_outputs.cuda()).loss
loss.backward()
optimizer.step()
optimizer.zero_grad()
```
My evaluation code is similar:
```auto
with torch.no_grad():
for target_tensor_input, t5_outputs, t5_masks in tqdm(train_dataloader):
# the output here varies
batch_outputs_generate = model.generate(input_ids=target_tensor_input.cuda(), min_length=2, max_length=max_target_length, do_sample=True, num_beams=8)
batch_outputs_generate = tokenizer.batch_decode(batch_outputs_generate, skip_special_tokens=True)
# the output here is good
batch_outputs_forward = model(input_ids=target_tensor_input.cuda(), attention_mask=t5_masks.cuda(), labels=t5_outputs.cuda())
batch_outputs_forward_output = tokenizer.batch_decode(torch.argmax(batch_outputs_forward.logits, dim=2).tolist(), skip_special_tokens=True)
```
Thanks!
---
**Author:**  [@hockeybro12](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhockeybro12)%5C
**Post date:** [September 11, 2022, 10:12pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-model-generate-and-model-outputs-vastly-different%2F21715%2F2 "2022-09-11T22:12:59Z")
Hello everyone, any update here? I’m Still confused on this topic.
---
**Author:**  [@AD-I](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAD-I)%5C
**Post date:** [August 19, 2025, 12:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-model-generate-and-model-outputs-vastly-different%2F21715%2F3 "2025-08-19T12:44:12Z")
I am currently working with t5 also, I never noticed something like this, but what i found is that sometimes the model gets empty outputs. I will check if generate function is the cause.
Can’t you just stick with the forward model outputs? What is the benefits to use generate function?