# Prohibitively large RAM consumption on Trainer validation

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fprohibitively-large-ram-consumption-on-trainer-validation%2F83486%3E%5C
**Category:** 🤗Transformers\
**Created:** [April 24, 2024, 8:12am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fprohibitively-large-ram-consumption-on-trainer-validation%2F83486 "2024-04-24T08:12:14Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![dmytromishkin](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dmytromishkin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdmytromishkin)%5C
**Post date:** [April 24, 2024, 8:12am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fprohibitively-large-ram-consumption-on-trainer-validation%2F83486%2F1 "2024-04-24T08:12:14Z")

</div>

That is not a question, rather an issue I wanted share, as it took me a while to debug.  
I am training an image segmentation model on a large dataset with Trainer.  
The images are rather big (1024x1024), I have many classes and and the validation set is 500 examples. Once I am doing validation (after the prediction step is over), machine freezes for 5-10 minutes and then either continue, or crashes for unknown reason.

After hours spent, I figured out the reason.

Here is my RAM consumption graph:

 ![GL6j10EX0AAOAAe](https://us1.discourse-cdn.com/hellohellohello/original/3X/4/6/46dc51845e8ddb41861e226f7b4ee83522b3607d.png)

The reasons for this is that Trainer first predicts ALL examples and then stores them.

> <https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L3697>

As you can imaging, creating 1024x1024x100x500 logits tensor and 1024x1024x500 labels tensor takes some memory.

I have passed `eval_do_concat_batches=False`, otherwise the iterated concat with every batch kill the performance and RAM even faster.

As far as I searched, there is no way of doing evaluation “per-sample” and then averaging the metrics, as it is common in computer vision. I hope this thread will save someone hours of debugging

---

<div class="post-metadata">

**Author:** ![dmytromishkin](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dmytromishkin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdmytromishkin)%5C
**Post date:** [April 24, 2024, 9:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fprohibitively-large-ram-consumption-on-trainer-validation%2F83486%2F2 "2024-04-24T21:48:35Z")

</div>

Solution in this PR

> <https://github.com/huggingface/transformers/pull/28769>
>
> \# What does this PR do?
> 
> This PR does two things which are necessary for using… the Trainer in resource constrained environments (like my RTX-3070Ti machine):
> 1. Add cache clearing in training and evaluation loops
> - This reduces peak GPU load and prevents CUDA OOM errors when running near capacity.
> 2. Add Trainer arg \`batch\_eval\_metrics\` for batched eval metrics computation.
> - When working with limited RAM, storing all logits across the entire evaluation set may not be feasible. A user working in this condition can pass \`True\` to \`batch\_eval\_metrics\` and construct a \`compute\_metrics\` function which can update average metrics at a batch level to prevent OOM errors with large eval sets. Particularly useful for vision transformers.
> - Previous functionality is unaltered if option is not set to \`True\`
> 
> @muellerzr

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [April 25, 2024, 9:49am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fprohibitively-large-ram-consumption-on-trainer-validation%2F83486%2F3 "2024-04-25T09:49:30Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.