Open-Source LLM Evaluation and Annotation QA Benchmark

Beginners
1

Hello Hugging Face folks,

I’ve developed an open-source Python project demonstrating a practical workflow for LLM response evaluation and annotation quality assurance.

The benchmark covers:

  • Weighted response scoring across four dimensions.

  • Structured annotation validation using Pydantic.

  • A shared error taxonomy.

  • Evaluator agreement and disagreement analysis.

  • Human adjudication for disputed cases.

The sample pipeline includes 15 shared evaluation cases, 86.7% error-label agreement, 93.3% severity agreement and 3 cases flagged for human review.

I’m interested in feedback from practitioners working on LLM evaluation, RLHF, data quality and annotation workflows.

Some questions I’d particularly appreciate feedback on:

  1. Which additional evaluation dimensions would you recommend?

  2. What is the best approach to extend this into RAG factuality and citation correctness evaluation?

  3. Would integration with Label Studio or Argilla be useful?

  4. What additional metrics should be included in evaluator agreement reporting?

Dataset: goodgentleman918/ai-data-quality-llm-evaluation-benchmark · Datasets at Hugging Face

GitHub: GitHub - Enterprise-AI-Solutions/AI-Data-Quality-LLM-Evaluation-Benchmark: A Python-based AI data quality and LLM response evaluation benchmark for rubric scoring, hallucination analysis, annotation QA, evaluator agreement, disagreement detection, and adjudication. · GitHub

Contributions and constructive suggestions are welcome.

2

I’d score RAG in two parts. First, retrieval: did the passages hold what the answer needed? Second, the answer itself. Split it into single claims and check each one against the retrieved text. The score is supported claims over total claims. Ragas defines its faithfulness metric this way.

Citation correctness is a separate check. For each claim, take only the passage it cites and ask if that passage alone backs it. An answer can be true and still cite the wrong source. The ALCE benchmark also keeps correctness and citation quality apart.

For agreement reporting, I’d have raters give each claim one label: supported and cited right, supported but cited wrong, or unsupported. Then you can report agreement per claim, not only per answer.