# Trainer doesn't show the loss at each step
**URL:**
**Author:**  [@pyantony](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpyantony)%5C
**Post date:** [July 30, 2020, 9:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-doesnt-show-the-loss-at-each-step%2F542%2F1 "2020-07-30T21:34:03Z")
Hi,
is there a way to display/print the loss (or metrics if you are evaluating) at each step (or n steps) or every time you log? I donโt see any option for that. This is very important causeโ it is the only way to tell if the model is learning or not. I thought โdebugโ was going to work but it seems to be deprecated. I am training in a jupyter notebook by the way.
Also, what about an early stopping option?
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [July 30, 2020, 9:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-doesnt-show-the-loss-at-each-step%2F542%2F2 "2020-07-30T21:59:34Z")
The loss and metrics are printed every `logging_steps` (there was w bug recently fixed, so you might need to update your install to an installation from source). As for early stopping, there is a PR under review with it, so it should come soon.
---
**Author:**  [@pyantony](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpyantony)%5C
**Post date:** [July 30, 2020, 10:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-doesnt-show-the-loss-at-each-step%2F542%2F3 "2020-07-30T22:22:12Z")
mmm, will try updating then. Thank you!
---
**Author:**  [@melody-ju](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmelody-ju)%5C
**Post date:** [September 3, 2020, 7:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-doesnt-show-the-loss-at-each-step%2F542%2F4 "2020-09-03T07:28:43Z")
Hi, I built from source yesterday but I still donโt think Iโm seeing the expected behavior when it comes to logging.
With gradient\_accumulation\_steps=16, logging\_steps=100 and eval\_steps=100, I expect to see both the loss and validation metrics printed at iteration 100 but nothing is printed at step 100.
With gradient\_accumulation\_steps=1, logging\_steps=100 and eval\_steps=100, only the loss and learning rate (no eval metrics) are printed once at step 100 and then at step 200 cuda runs out of memory. (With the prev config gradient\_accumulation\_steps=16, logging\_steps=100 and eval\_steps=100, the memory crash doesnโt happen).
In addition I canโt seem to find any of the train or eval metrics in the tf event files. Is there anything that needs to be done explicitly to log info to tensorboard?
I added my compute metrics function below as well as where Iโm instantiating the training arguments and trainer.
Thank you!
```
def compute_metrics(pred):
labels_ids = pred.label_ids
pred_ids = pred.predictions
# all unnecessary tokens are removed
pred_str = tokenizer.batch_decode(pred_ids, skip_special_tokens=True)
label_str = tokenizer.batch_decode(labels_ids, skip_special_tokens=True)
rouge_output = rouge.compute(predictions=pred_str, references=label_str, rouge_types=["rouge2"])["rouge2"].mid
return {
"rouge2_precision": round(rouge_output.precision, 4),
"rouge2_recall": round(rouge_output.recall, 4),
"rouge2_fmeasure": round(rouge_output.fmeasure, 4),
}
```
* * *
```
training_args = TrainingArguments(
output_dir="./",
per_device_train_batch_size=batch_size,
per_device_eval_batch_size=batch_size,
evaluate_during_training=True,
do_train=True,
do_eval=True,
# fp16=True, # This has a known bug with t5
gradient_accumulation_steps=16,
logging_steps=100,
eval_steps=100,
overwrite_output_dir=True,
save_total_limit=10,
)
trainer = Trainer(
model=model,
args=training_args,
compute_metrics=compute_metrics,
train_dataset=train_dataset,
eval_dataset=valid_dataset
)
```
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [September 3, 2020, 11:14am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-doesnt-show-the-loss-at-each-step%2F542%2F5 "2020-09-03T11:14:11Z")
So the explanation of the first point is that right now, your eval\_steps and logging steps have to be a round multiple of your gradient accumulation steps since those are tested only when you actually do an update (might fix that in the future but I have to think of it more). Since 100 is not a round multiple of 16, you canโt see anything.
In the second test, the metrics should be logged at step 100, so this one is weird. If you donโt see them printed, itโs logical you donโt see them in tensorboard either, it just means they were never logged.
---
**Author:**  [@melody-ju](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmelody-ju)%5C
**Post date:** [September 3, 2020, 1:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-doesnt-show-the-loss-at-each-step%2F542%2F6 "2020-09-03T13:06:05Z")
Hi, thanks for your reply!
Coming back to 100 not being a round mulitple of 16 I changed the gradient accumulation steps to 32 and logging steps to 128. I would expect the logging to happen then at 128th step, but nothing is printed. Any insight?
---
**Author:**  [@melody-ju](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmelody-ju)%5C
**Post date:** [September 3, 2020, 2:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-doesnt-show-the-loss-at-each-step%2F542%2F7 "2020-09-03T14:25:28Z")
Hmmm I want to add that although save\_steps is 512 there has been nothing written to the specified checkpoint output dir. (Now it is on step ~2000, and still nothing printed for logging either).
Showing my full training args below.
Any insight would be greatly appreciated. Iโm really scratching my head over the logging and saving issue.
```
batch_size = 1
training_args = TrainingArguments(
output_dir="./checkpoints",
per_device_train_batch_size=batch_size,
do_train=True,
# fp16=True, # This has a known bug with t5
gradient_accumulation_steps=32,
logging_steps=128,
save_steps=512,
overwrite_output_dir=True,
save_total_limit=10,
)
optimizer = Adafactor(model.parameters(), lr=1e-3, relative_step=False, warmup_init=False)
scheduler = get_constant_schedule(optimizer)
optimizers = optimizer, scheduler
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset,
optimizers=optimizers
)
trainer.train()
```
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [September 3, 2020, 8:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-doesnt-show-the-loss-at-each-step%2F542%2F9 "2020-09-03T20:37:55Z")
Ah, sorry I misread the code. When doing gradient accumulation, one step is one backward pass, so appends every gradient\_accumulation\_steps examples. So you need to lower your logging\_steps/eval\_steps/save\_steps to see something happening.
---
**Author:**  [@melody-ju](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmelody-ju)%5C
**Post date:** [September 4, 2020, 9:18am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-doesnt-show-the-loss-at-each-step%2F542%2F10 "2020-09-04T09:18:38Z")
Thanks, that makes sense ๐
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [September 4, 2020, 12:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-doesnt-show-the-loss-at-each-step%2F542%2F11 "2020-09-04T12:46:03Z")
I made a PR to add a warning about this in the documentation, so that other users are not surprised ๐
---
**Author:**  [@poli](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpoli)%5C
**Post date:** [September 30, 2020, 9:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-doesnt-show-the-loss-at-each-step%2F542%2F12 "2020-09-30T21:53:05Z")
I have the same problem. I donโt see the loss reported. Using version 3.0.2. Here is my code:
training\_args = TrainingArguments(
output\_dir="./ClaimsBERTo\_WordLevel",
overwrite\_output\_dir=False,
num\_train\_epochs=3,
per\_gpu\_train\_batch\_size=64,
save\_steps=10,
logging\_steps=10,
eval\_steps=10,
save\_total\_limit=5,
evaluate\_during\_training=True,
do\_eval=True,
logging\_dir=โ./logsโ,
)
trainer = Trainer(
model=model,
args=training\_args,
data\_collator=data\_collator,
train\_dataset=train\_dataset,
eval\_dataset=val\_dataset,
prediction\_loss\_only=True,
)
I hope someone will be able to help.
Thank you
---
**Author:**  [@poli](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpoli)%5C
**Post date:** [October 1, 2020, 4:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-doesnt-show-the-loss-at-each-step%2F542%2F14 "2020-10-01T16:20:56Z")
I am not sure it was done correctly but once I added the following: logging.basicConfig(level=logging.DEBUG,
format=โ%(asctime)s %(message)sโ)
to the trainer.py I could see the losses printed.
---
**Author:**  [@brando](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbrando)%5C
**Post date:** [August 16, 2022, 4:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-doesnt-show-the-loss-at-each-step%2F542%2F15 "2022-08-16T16:38:40Z")
@sgugger @poli I canโt see the train loss, thats what I want to see:
```auto
{'eval_loss': 2.9247915744781494, 'eval_runtime': 0.0168, 'eval_samples_per_second': 595.866, 'eval_steps_per_second': 59.587, 'epoch': 2822.0}
28%|โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
```
---
**Author:**  [@SUNM](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSUNM)%5C
**Post date:** [May 19, 2023, 3:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-doesnt-show-the-loss-at-each-step%2F542%2F16 "2023-05-19T03:52:27Z")
Hi @brando , how are u doing? sorry I wana see training and validation loss. this is my code. I wana see the logs with tensorboard. do you have any idea how it is possible?
```auto
training_args = TrainingArguments(output_dir=Results_Path, learning_rate=5e-5,num_train_epochs=15, evaluation_strategy="epoch", logging_strategy="epoch",save_strategy="epoch",seed=42,load_best_model_at_end=True,logging_steps=5000,
per_device_train_batch_size=2, save_total_limit=1,per_device_eval_batch_size=2,warmup_steps=100, weight_decay=0.01, logging_dir=Results_Path)
Trainer(model=model, args=training_args,train_dataset=train_dataset,
eval_dataset=val_dataset,data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]),
'attention_mask': torch.stack([f[1] for f in data]),
'labels': torch.stack([f[0] for f in data])}).train()
```
---
**Author:**  [@ivanzhouyq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fivanzhouyq)%5C
**Post date:** [July 7, 2023, 8:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-doesnt-show-the-loss-at-each-step%2F542%2F17 "2023-07-07T20:46:25Z")
I encountered the same problem โ I cannot see train loss or throughput. Only eval loss and samples per second is shown:
```auto
{'eval_loss': 8.6328125, 'eval_runtime': 26.1463, 'eval_samples_per_second': 17.134, 'eval_steps_per_second': 0.535, 'epoch': 0.71}
```
---
**Author:**  [@lokesh005](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flokesh005)%5C
**Post date:** [October 4, 2023, 3:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-doesnt-show-the-loss-at-each-step%2F542%2F18 "2023-10-04T15:38:13Z")
I think you just need to scroll through the nine panels available (only 6 out of 9 displayed on your screenshot) and youโll find the actual training loss under `train/loss` and not `train/train_loss`
---
**Author:**  [@chen806](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fchen806)%5C
**Post date:** [October 11, 2023, 3:16am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-doesnt-show-the-loss-at-each-step%2F542%2F19 "2023-10-11T03:16:39Z")
I am not sure if it is jupyter or hf trainer issue. I try to play around with a lot of combinations of eval\_steps, evaluation\_strategy, logging steps. They do not work as expected. Though i am setting logging\_steps = 50. I am still seeing loss every 10 steps and eval\_steps does not work either. Anyone knows why?
---
**Author:**  [@fulltrend](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffulltrend)%5C
**Post date:** [November 30, 2023, 8:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-doesnt-show-the-loss-at-each-step%2F542%2F20 "2023-11-30T20:37:19Z")
This is coming late to the game, but in case someone else still struggles - you may need to provide a valid **report\_to** param, such as:
**`report_to='tensorboard'`**
so that the trainer knows how to format the output.
---
**Author:**  [@kkruskal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkkruskal)%5C
**Post date:** [January 3, 2024, 11:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-doesnt-show-the-loss-at-each-step%2F542%2F21 "2024-01-03T11:53:44Z")
Iโve found that if you use JupyterNotebook in Pycharm it causes this problem, running it in a browser everything works fine.
---
**Author:**  [@SCUER](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSCUER)%5C
**Post date:** [January 24, 2024, 1:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-doesnt-show-the-loss-at-each-step%2F542%2F22 "2024-01-24T13:15:06Z")
i have been use notebook in pycharm๏ผi will try
[Next page](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-doesnt-show-the-loss-at-each-step%2F542.md%3Fpage%3D2)