# How to get per-eval-step score when using trainer?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-get-per-eval-step-score-when-using-trainer%2F4069%3E%5C
**Category:** 🤗Transformers\
**Created:** [March 2, 2021, 6:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-get-per-eval-step-score-when-using-trainer%2F4069 "2021-03-02T06:10:11Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![mralexis](https://avatars.discourse-cdn.com/v4/letter/m/ecb155/32.png) [@mralexis](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmralexis)%5C
**Post date:** [March 2, 2021, 6:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-get-per-eval-step-score-when-using-trainer%2F4069%2F1 "2021-03-02T06:10:11Z")

</div>

I am training a model and am hoping to get the log of the per-evaluation-step performance while training. I currently set `--evaluation_strategy steps` and I could see logs got printed. However, is there a way I could get the table as an object (json or pd.DataFrame) in Python?

```auto
|Step|Training Loss|Validation Loss|Rouge1|Rouge2|Rougel|Rougelsum|Gen Len|Runtime|Samples Per Second|
|---|---|---|---|---|---|---|---|---|---|
|100|No log|2.458449|39.987500|16.891800|37.788500|37.794300|10.292600|26.358400|44.730000|
|200|No log|2.128455|49.043000|24.468600|47.341900|47.270600|10.463100|26.245500|44.922000|
|300|No log|1.980806|51.324400|25.405300|49.549300|49.507100|10.305300|25.733800|45.815000|
|400|No log|1.892222|53.523700|27.361200|51.650900|51.613200|10.371500|25.708300|45.861000|
|500|0.997900|1.840045|54.044500|27.843400|52.146600|52.125500|10.392700|25.551800|46.142000|
|600|0.997900|1.810891|54.251600|28.728500|52.545900|52.492900|10.435100|26.443400|44.586000|
|700|0.997900|1.799217|54.409300|28.685000|52.649800|52.611400|10.400300|26.362800|44.722000|

```

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flewtun)%5C
**Post date:** [March 2, 2021, 8:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-get-per-eval-step-score-when-using-trainer%2F4069%2F2 "2021-03-02T08:00:22Z")

</div>

Hi @mralexis, you could write a simple callback that saves the logs to disk - e.g. by adapting the `PrinterCallback`: [transformers.trainer\_callback — transformers 4.3.0 documentation](https://huggingface.co/transformers/_modules/transformers/trainer_callback.html#PrinterCallback)

You can then pass your callback to the `Trainer` with the `callbacks` argument. Then you can load and process them after training 😄

---

<div class="post-metadata">

**Author:** ![mralexis](https://avatars.discourse-cdn.com/v4/letter/m/ecb155/32.png) [@mralexis](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmralexis)%5C
**Post date:** [March 2, 2021, 8:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-get-per-eval-step-score-when-using-trainer%2F4069%2F3 "2021-03-02T08:15:09Z")

</div>

Thanks, @lewtun !

Out of curiosity, how does the `load_best_model_at_end` parameter work? If it loads the best model based on the x times evaluation over training, then there must be something like a `best_step` stored somewhere?

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flewtun)%5C
**Post date:** [March 2, 2021, 10:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-get-per-eval-step-score-when-using-trainer%2F4069%2F4 "2021-03-02T10:10:55Z")

</div>

Hi @mralexis, as described in the [docs](https://huggingface.co/transformers/main_classes/trainer.html?highlight=trainer#trainingarguments), `load_best_model_at_end` works in conjunction with `metric_for_best_model`.

Under the hood, the `Trainer` keeps track of the best checkpoint seen to date via the `Trainer.state.best_model_checkpoint` attribute (see [here](https://github.com/huggingface/transformers/blob/0234de8418b253e843dda0a18a3e18476de52781/src/transformers/trainer.py#L1237)) which is then used by `load_best_model_at_end` [here](https://github.com/huggingface/transformers/blob/0234de8418b253e843dda0a18a3e18476de52781/src/transformers/trainer.py#L1125).

---

<div class="post-metadata">

**Author:** ![hugjoop](https://avatars.discourse-cdn.com/v4/letter/h/b4bc9f/32.png) [@hugjoop](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhugjoop)%5C
**Post date:** [October 18, 2022, 10:42am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-get-per-eval-step-score-when-using-trainer%2F4069%2F5 "2022-10-18T10:42:20Z")

</div>

@lewtun Just to be sure, so the `Trainer` object does not keep track of the metrics such that we as end-users can access them as well? Writing a callback for this feels a bit cumbersome, as all we want is already displayed every step/epoch by the `Trainer` object.

I do not know how hugging face stores these metrics internally, but to me it seems the most logical that it keeps the metrics in memory somewhere and re-displays the entire dataframe after each step. But I could be wrong here.