# How to evaluate before first training step?
**URL:**
**Author:**  [@johngiorgi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjohngiorgi)%5C
**Post date:** [June 8, 2022, 2:44am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-evaluate-before-first-training-step%2F18838%2F1 "2022-06-08T02:44:34Z")
Hi! I have a use case where I would like evaluation to happen _at the beginning of training_ (before a training step has been taken) in addition to every n steps. I can easily get the latter using [`evaluation_strategy`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.evaluation_strategy) and [`eval_steps`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.eval_steps), but not sure how to get the former.
There is an argument, [`logging_first_step`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.logging_first_step) that sounds like it should do _exactly_ what I need:
> **logging\_first\_step** (`bool`, _optional_, defaults to `False`) â Whether to log and evaluate the first `global_step` or not.
But providing this argument does not lead to evaluation at the beginning of training like I would expect (tested with the [`run_summarization.py`](https://github.com/huggingface/transformers/blob/main/examples/pytorch/summarization/run_summarization.py) script). Does anyone have an idea how to get this behaviour from the HF [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer#trainer)?
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [June 8, 2022, 11:36am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-evaluate-before-first-training-step%2F18838%2F2 "2022-06-08T11:36:33Z")
You can just add a line `trainer.evaluate()` before the call to `train`.
---
**Author:**  [@johngiorgi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjohngiorgi)%5C
**Post date:** [June 8, 2022, 4:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-evaluate-before-first-training-step%2F18838%2F3 "2022-06-08T16:14:58Z")
Thanks for the quick response! I guess this solution works, but the results do not end up in the `log_history` field of the `training_state.json` file, which is how I am tracking performance over time. So I guess two questions:
- Is there a better way to track evaluation metrics over time (using the provided example scripts like `run_summarization.py`) than the `log_history` field of the `training_state.json`?
- Might adding this functionality (evaluating during training _before_ a training step has been taken) be a good idea? Seems like a pretty ubiquitous use case to me as you may want to plot performance over time and knowing the performance of the model (either randomly initialized or pre-trained) before the first train step is useful. Would be happy to take a crack at this if I could get some advice as to where to implement it.
---
**Author:**  [@frankier](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffrankier)%5C
**Post date:** [January 17, 2023, 2:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-evaluate-before-first-training-step%2F18838%2F4 "2023-01-17T14:41:19Z")
A alternative approach, which may or may not solve the problem is to use a callback, like so:
```auto
class EvaluateFirstStepCallback(TrainerCallback):
def on_step_end(self, args, state, control, **kwargs):
if state.global_step == 1:
control.should_evaluate = True
trainer.add_callback(EvaluateFirstStepCallback())
```
It does seem like it might be nice to have as a built-in TrainingArguments, given it mirrors `logging_first_step` pretty closely.
---
**Author:**  [@kwang2049](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkwang2049)%5C
**Post date:** [August 4, 2023, 11:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-evaluate-before-first-training-step%2F18838%2F5 "2023-08-04T11:27:19Z")
THanks for the post. One small issue here is, by doing this way, there will be one training step already done, which is not equal to the model evaluation without any training đ¤
---
**Author:**  [@zouharvi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzouharvi)%5C
**Post date:** [August 31, 2023, 2:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-evaluate-before-first-training-step%2F18838%2F6 "2023-08-31T14:58:08Z")
Doesnât that get solved using `on_step_begin`? Works for me for NLLB finetuning with `Seq2SeqTrainingArguments`. The only caveat seems to be that train loss doesnât exist at this point so the wandb plots are offset.
```auto
class EvaluateFirstStepCallback(TrainerCallback):
def on_step_begin(self, args, state, control, **kwargs):
if state.global_step == 1:
control.should_evaluate = True
trainer.add_callback(EvaluateFirstStepCallback())
```
---
**Author:**  [@tongyx361](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftongyx361)%5C
**Post date:** [September 23, 2023, 7:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-evaluate-before-first-training-step%2F18838%2F7 "2023-09-23T19:41:20Z")
The code provided by @frankier and @zouharvi both seem to have some minor errors
According to [trainer.py](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L1867-L1980)
- `on_step_begin()`: `if step % args.gradient_accumulation_steps == 0:`, called before all operations except for the random state, when **`global_step` has not yet been updated, and the model parameters have not yet been updated**
- `on_step_end()`: called after all operations in each training iteration, when `global_step` has been updated and **when `global_step == 1`, if the gradient accumulation step is 1, the model parameters have been updated once**
Therefore, it is necessary to **call `on_step_begin()` at `global_step == 0`** in order to ensure that model parameters **that have not yet been updated** are evaluated.
The corresponding code is below:
```plaintext
class EvaluateFirstStepCallback(TrainerCallback):
def on_step_begin(self, args, state, control, **kwargs):
if state.global_step == 0:
control.should_evaluate = True
trainer.add_callback(EvaluateFirstStepCallback())
```
---
**Author:**  [@wdavies](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwdavies)%5C
**Post date:** [February 17, 2024, 10:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-evaluate-before-first-training-step%2F18838%2F8 "2024-02-17T22:35:09Z")
I wish theyâd fix this in the actual code. Itâs so annoying that the flag for logging delay doesnât work as expected. Wasted 10-15 mins finding this solution.
---
**Author:**  [@jacklanda](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjacklanda)%5C
**Post date:** [April 25, 2024, 10:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-evaluate-before-first-training-step%2F18838%2F9 "2024-04-25T10:53:56Z")
I have to say the progress bar will also step to â1â even using your providing code:
---
**Author:**  [@mmordig](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmmordig)%5C
**Post date:** [July 31, 2024, 5:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-evaluate-before-first-training-step%2F18838%2F10 "2024-07-31T05:27:14Z")
The reason is that evaluation happens after training, i.e. should\_evaluate is only checked afterwards.
---
**Author:**  [@jacklanda](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjacklanda)%5C
**Post date:** [August 3, 2024, 4:12pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-evaluate-before-first-training-step%2F18838%2F11 "2024-08-03T16:12:12Z")
Hey everybody!
Now (2024) we can pass the parameter `eval_on_start` to your initialized `TrainingArguments` object to make your model evaluate before undergoing any training steps đ
This new parameter was renamed from a deprecated one, âsanity\_evaluationâ, as introduced [here](https://github.com/huggingface/transformers/pull/31192).
I hope this update will find all of you well!