# Compute\_metrics() behaves strangely in distributed setting

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-metrics-behaves-strangely-in-distributed-setting%2F99804%3E%5C
**Category:** 🤗Transformers\
**Created:** [July 28, 2024, 6:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-metrics-behaves-strangely-in-distributed-setting%2F99804 "2024-07-28T18:19:14Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![piggyman](https://avatars.discourse-cdn.com/v4/letter/p/9de0a6/32.png) [@piggyman](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpiggyman)%5C
**Post date:** [July 28, 2024, 6:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-metrics-behaves-strangely-in-distributed-setting%2F99804%2F1 "2024-07-28T18:19:14Z")

</div>

Hello!

I am using the HF Seq2SeqTrainer to fine-tune CodeT5 on a task. I run via torchrun and run on a single node with 2 GPUs.

I recently noticed that compute\_metrics() function I wrote receives EvalPredictions instance with more sequences than there are in my validation set. The val set I provide to the Trainer is of size 250; whereas if you add

> print(preds.shape)

into your compute\_metrics() function, it prints ( **264** , \*) where \* is variable depending on the longest prediction the model comes up with.

Upon further inspection, the excess [250,264) entries **are just repeats of [1,14)?** This issue disappears completely if I run without parallelisation.

Also, compute\_metrics() seems to be getting called from both devices as the .shape information is printed twice (once from rank 0 and once from rank 1).

Can anyone suggest what is happening? ☹