# How to evaluate CLMs on MMLU?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-evaluate-clms-on-mmlu%2F54564%3E%5C
**Category:** Beginners\
**Created:** [September 12, 2023, 1:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-evaluate-clms-on-mmlu%2F54564 "2023-09-12T01:05:48Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![surya-narayanan](https://avatars.discourse-cdn.com/v4/letter/s/d2c977/32.png) [@surya-narayanan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsurya-narayanan)%5C
**Post date:** [September 12, 2023, 1:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-evaluate-clms-on-mmlu%2F54564%2F1 "2023-09-12T01:05:49Z")

</div>

Hi folks,

I’m trying to evaluate LLMs using a pipeline on the cais/mmlu dataset. Right now, I am merging the options with the questions and passing the merged Question + Options as an input to an LLM wrapped in a pipeline API. However, there are two problems associated with this workflow.

1. The LLMs are extremely slow (take \> 60 s for a single query for a 13B model on 8x A6000 GPUs)
2. Model outputs don’t always match exactly to one of the options in the multiple-choice option set.

How can I overcome these issues? Does anyone have any suggestions?