A-Eval: What is the best model? Application-driven Evaluation for Large Language Models (NLPCC 2024 Accepted)

This project is a sub-project of UnicomAI: https://github.com/UnicomAI/UnicomBenchmark/tree/main/A-Eval

Introduction

A-Eval is a benchmark designed to evaluate Chat LLMs of various scales from a practical application perspective. The dataset includes 678 question-and-answer pairs spanning 5 categories, 27 sub-categories, and 3 difficulty levels. A-Eval offers clear empirical and engineering guidelines for choosing the "best" model for real-world applications.

Application-driven tasks taxonomy

• 678 QA pairs
• 5 categories, 27 sub-categories
• 3 difficulty levels

Evaluation Results

Based on QWen1.5-72B-Chat, we design an automatic evaluation method to evaluate models of 8 varying scales. Our additional expert evaluation validates the reliability of the automatic evaluation method.

Average accuracy

We presents the average accuracy of models with different scales on A-Eval.

(a) The average accuracy of models of varying scales across all tasks and difficulty levels. The dotted line represents expert evaluation results, while the solid lines represent automatic evaluation results with different scoring thresholds T.

(b) The average accuracy of models of varying scales on easy, medium, and hard data. The dotted line represents expert evaluation results, and the solid lines depict automatic evaluation results using scoring thresholds of 90 and 60, respectively.

Accuracy by task

For each specific task and its corresponding sub-tasks, the average accuracy of models with different scales.

(a) Accuracy when T = 60. (b) Accuracy when T = 90.

Model Selection

The best model is defined as the one that achieves the desired accuracy with the smallest size. Using the evaluation results, users can effortlessly identify the best model by drawing horizontal lines on the performance charts. This is an example:

TODO

We will provide code to evaluate the model using A-Eval

Citation

If you use our benchmark or dataset in your research, please cite our paper.

@misc{lian2024best,
      title={What is the best model? Application-driven Evaluation for Large Language Models}, 
      author={Shiguo Lian and Kaikai Zhao and Xinhui Liu and Xuejiao Lei and Bikun Yang and Wenjing Zhang and Kai Wang and Zhaoxiang Liu},
      year={2024},
      eprint={2406.10307},
      archivePrefix={arXiv},
}