# KeyError: 'eval\_accuracy' when running trainer
**URL:**
**Author:**  [@GBaker](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGBaker)%5C
**Post date:** [March 5, 2023, 12:16am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkeyerror-eval-accuracy-when-running-trainer%2F33050%2F1 "2023-03-05T00:16:11Z")
Trying to train a model on the ‘medmcqa’ dataset, but the trainer throws an error after the first epoch.
Here is my trainer code:
```auto
training_args = TrainingArguments(
output_dir="biolinkbert-base-mcmqa",
label_names=["cop"],
evaluation_strategy="epoch",
save_strategy="epoch",
load_best_model_at_end=True,
metric_for_best_model="accuracy",
learning_rate=3e-6,
fp16=True,
per_device_train_batch_size=32,
per_device_eval_batch_size=32,
gradient_accumulation_steps=32,
num_train_epochs=30,
seed=42,
weight_decay=1e-3,
push_to_hub=True,
logging_strategy="epoch",
eval_delay=0,
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=tokenized_dataset["train"],
eval_dataset=tokenized_dataset["validation"],
tokenizer=tokenizer,
data_collator=DataCollatorForMultipleChoice(tokenizer=tokenizer),
compute_metrics=compute_metrics,
callbacks=[early_stopper],
)
trainer.train()
```
compute\_metrics:
```auto
import evaluate
accuracy = evaluate.load("accuracy")
def compute_metrics(eval_pred):
predictions, labels = eval_pred
predictions = np.argmax(predictions, axis=1)
return accuracy.compute(predictions=predictions, references=labels)
```
Any help is greatly appreciated!
---
**Author:**  [@prit-sk](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fprit-sk)%5C
**Post date:** [March 5, 2023, 10:21am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkeyerror-eval-accuracy-when-running-trainer%2F33050%2F2 "2023-03-05T10:21:04Z")
use this:
```auto
import evaluate
metric = evaluate.load("accuracy")
def compute_metrics(eval_pred):
predictions, labels = eval_pred
predictions = np.argmax(predictions, axis=1)
return metric.compute(predictions=predictions, references=labels)
```
Try the above code.
---
**Author:**  [@GBaker](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGBaker)%5C
**Post date:** [March 5, 2023, 2:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkeyerror-eval-accuracy-when-running-trainer%2F33050%2F3 "2023-03-05T14:53:39Z")
That didn’t appear to work.
After placing a print statement in the compute\_metrics function it seems that it isn’t ever being called.
---
**Author:**  [@prit-sk](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fprit-sk)%5C
**Post date:** [March 5, 2023, 4:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkeyerror-eval-accuracy-when-running-trainer%2F33050%2F4 "2023-03-05T16:41:35Z")
i also got the same error, but i am not able to recall how i solved it. but here are few observations from your code, try all of this:
1.
```auto
from datasets import load_metric
metric = load_metric("accuracy") inplace of accuracy = evaluate.load("accuracy")
def compute_metrics(eval_pred):
predictions, labels = eval_pred
predictions = np.argmax(predictions, axis=1)
return metric.compute(predictions=predictions, references=labels)
```
1. any one is needed from the below.
```auto
tokenizer=tokenizer,
data_collator=DataCollatorForMultipleChoice(tokenizer=tokenizer)
```
i think there is a package version issue of either transformers or evaluate. Not sure though.
anyway try the above. Also why there is only single label name in your code? I didn’t get that
---
**Author:**  [@GBaker](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGBaker)%5C
**Post date:** [March 5, 2023, 5:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkeyerror-eval-accuracy-when-running-trainer%2F33050%2F5 "2023-03-05T17:00:04Z")
I’ll try those things, thanks. Just strange because I’ve used the code pretty much as it is with another multiple choice dataset recently and it worked fine.
The label name “cop” corresponds to the dataset column containing an integer from 0-3 representing the correct multiple choice answer.
---
**Author:**  [@NikhilKhodake](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNikhilKhodake)%5C
**Post date:** [March 6, 2023, 7:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkeyerror-eval-accuracy-when-running-trainer%2F33050%2F6 "2023-03-06T07:38:04Z")
I am also getting this error- “TypeError: only size-1 arrays can be converted to Python scalars”. Basically I am fine-tunning the facebook/bart-base model, with samsum Dataset, in Amazon sagemaker.
---
**Author:**  [@arastu-mudgal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Farastu-mudgal)%5C
**Post date:** [May 22, 2023, 8:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkeyerror-eval-accuracy-when-running-trainer%2F33050%2F7 "2023-05-22T08:05:13Z")
any fix for this yet?
---
**Author:**  [@divyaswamy87](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdivyaswamy87)%5C
**Post date:** [May 25, 2023, 2:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkeyerror-eval-accuracy-when-running-trainer%2F33050%2F8 "2023-05-25T14:38:00Z")
I get the same error - any fix to this issue
---
**Author:**  [@Guldeniz](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGuldeniz)%5C
**Post date:** [May 26, 2023, 10:59am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkeyerror-eval-accuracy-when-running-trainer%2F33050%2F9 "2023-05-26T10:59:47Z")
Hello.
I don’t know if it is the problem but in your compute\_metrics, this line;
`predictions = np.argmax(predictions, axis=1)`
Shouldn’t it be `axis=-1`?
---
**Author:**  [@David-Saeteros6](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDavid-Saeteros6)%5C
**Post date:** [October 5, 2023, 9:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkeyerror-eval-accuracy-when-running-trainer%2F33050%2F10 "2023-10-05T09:11:44Z")
This worked for me:
```auto
# Load metric
metric_name = "f1"
metric = load_metric(metric_name)
# Define metrics
def compute_metrics(eval_pred):
predictions, labels = eval_pred
predictions = np.argmax(predictions, axis=1)
# 'micro', 'macro', etc. are for multi-label classification. If you are running a binary classification, leave it as default or specify "binary" for average
return metric.compute(predictions=predictions, references=labels, average="binary")
```
---
**Author:**  [@dreidizzle](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdreidizzle)%5C
**Post date:** [October 8, 2023, 5:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkeyerror-eval-accuracy-when-running-trainer%2F33050%2F11 "2023-10-08T17:09:19Z")
For the people where this was working: What version are you using? I’m on transformers 4.32.0 and this does not work.