# Evaluation and compute\_metrics slowdown

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fevaluation-and-compute-metrics-slowdown%2F52682%3E%5C
**Category:** Intermediate\
**Created:** [August 29, 2023, 3:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fevaluation-and-compute-metrics-slowdown%2F52682 "2023-08-29T15:14:57Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![johannes-garstenauer](https://avatars.discourse-cdn.com/v4/letter/j/c6cbf5/32.png) [@johannes-garstenauer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjohannes-garstenauer)%5C
**Post date:** [August 29, 2023, 3:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fevaluation-and-compute-metrics-slowdown%2F52682%2F1 "2023-08-29T15:14:57Z")

</div>

I am trying to extract embeddings from a DistilBertModel and therefore need to perform inference on it with a large dataset.

When performing inference using `predict()` on DistilBertForSequenceClassification either CUDA out of memory or a significant slowdown occurs.

As suggested in other posts ([Similar Forum Post](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-out-of-memory-when-using-trainer-with-compute-metrics%2F2941)) I used `eval_accumulation_steps` to avoid the OOM issue, which however still leads to an extreme slowdown. Varying `eval_accumulutation_steps` values (5, 500, 1k) and `batch_sizes` did not seem to resolve the issue.

Another suggested approach was using `preprocess_logits_for_metrics`, however, this would also only partially decrease the slowdown. The only real solution was using the function to `return torch.tensor(0)`. This is a nonsensical return value but proves that there are some underlying memory issues.

```auto
def preprocess_logits_for_metrics_inference(logits, labels):
    """
    Original Trainer may have a memory leak.
    This is a workaround to avoid storing too many tensors that are not needed.
    """

    last_hidden_state = logits[1][0]
    output = last_hidden_state
    return output

```

Also using `prediction_loss_only=True` avoids this error indicating that the there is a memory error when processing the outputs during inference/evaluation.

Below you can find the full training and eval code. How come applying both `eval_accumulation_steps` and `preprocess_logits_for_metrics` cannot resolve the issue?

```auto
model = DistilBertForSequenceClassification.from_pretrained("johannes-garstenauer/distilbert-heaps-masked",
                                                            num_labels=2,
                                                            id2label=id2label, label2id=label2id)

from transformers import DataCollatorWithPadding, TrainingArguments, Trainer

data_collator = DataCollatorWithPadding(tokenizer=pretrained_tokenizer)

model_name = "bin_clean_seq_class"
print(f"Model name: {model_name}")

training_args = TrainingArguments(
    output_dir=model_name,
    learning_rate=2e-5,
    per_device_train_batch_size=64,
    per_device_eval_batch_size=64,
    num_train_epochs=3,
    weight_decay=0.01,
    evaluation_strategy="epoch",
    eval_accumulation_steps=500,
)

def preprocess_logits_for_metrics(logits, labels):
    """
    Original Trainer may have a memory leak.
    This is a workaround to avoid storing too many tensors that are not needed.
    """

    last_hidden_state = logits[1][0]
    output = last_hidden_state
    return output

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset_split["train"],
    eval_dataset=tokenized_dataset_split["test"],
    tokenizer=pretrained_tokenizer,
    data_collator=data_collator,
    preprocess_logits_for_metrics=preprocess_logits_for_metrics
)

pred = trainer.predict(test_dataset=tokenized_dataset_split["test"])

```