# Evaluating on MMLU while finetuning using Trainer

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fevaluating-on-mmlu-while-finetuning-using-trainer%2F57220%3E%5C
**Category:** 🤗Transformers\
**Created:** [October 3, 2023, 9:47am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fevaluating-on-mmlu-while-finetuning-using-trainer%2F57220 "2023-10-03T09:47:29Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![imdatta0](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/imdatta0/32/19610_2.png) [@imdatta0](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fimdatta0)%5C
**Post date:** [October 3, 2023, 9:47am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fevaluating-on-mmlu-while-finetuning-using-trainer%2F57220%2F1 "2023-10-03T09:47:29Z")

</div>

Hi all, I’m trying to finetune llama on custom text dataset and would want to check how it’d perform if I evaluated it on MMLU dataset. Here’s what I’m using rn ( the eval\_data and train\_data are subsets of custom dataset).

```auto
trainer = Trainer(
    model = model,
    train_dataset = train_data,
    eval_dataset = eval_data,
    args = **training_args
    data_collator=DataCollatorForLanguageModeling(tokenizer,pad_to_multiple_of=8, mlm=False)
)

```

If I replace eval\_data with MMLU dataset (aka by doing  
`eval_data = load_datset('mmlu','abstract_algebra')`  
), how do I tell the trainer to use Binary loss ( 1 or 0) instead of Cross Entropy loss (this is what it’d use normally ig)?