# Evaluate model at saved checkpoint

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fevaluate-model-at-saved-checkpoint%2F7068%3E%5C
**Category:** 🤗Transformers\
**Created:** [June 22, 2021, 2:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fevaluate-model-at-saved-checkpoint%2F7068 "2021-06-22T14:53:35Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![MillyXXX](https://avatars.discourse-cdn.com/v4/letter/m/a9a28c/32.png) [@MillyXXX](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMillyXXX)%5C
**Post date:** [June 22, 2021, 2:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fevaluate-model-at-saved-checkpoint%2F7068%2F1 "2021-06-22T14:53:36Z")

</div>

Hi, I’m using Seq2seq Trainer for QG and trying to see sample output of model at a checkpoint like so:

model = ProphetNetForConditionalGeneration.from\_pretrained(‘squad\_training\_latest/results/checkpoint-212000’)  
…  
contexts = [tdata[‘context’][i]]  
answers = [tdata[‘answer’][i]]  
questions = [tdata[‘question’][i], tdata[‘question2’][i]]

```
    encoder_inputs, decoder_inputs = preprocess_batch(contexts, questions, answers)
    decoder_inputs = decoder_inputs.contiguous()

    question_ids = model.generate(encoder_inputs, early_stopping=False, return_dict_in_generate=False, eos_token_id=102, min_length = 64)
    rv = tokenizer.batch_decode(question_ids, skip_special_tokens=False)
    print(rv)

```

The result looks very bad, overfitted like [’[SEP] what was the the the the the name of the? [X\_SEP] the the? name of the?’] for every example

however the eval loss was low so wondering if I’m evaluating right, or should I use Trainer.predict? load the model a different way?

(side question: I noticed Trainer.evaluate() returns teacher forced predictions to me and it differs from model.generate() and was wondering why and whether the param predict\_with\_generate has anything to do with it)