Results of the summarizing process for the arXiv paper: 2306.05179v1

This study introduces M3Exam, a novel benchmark for evaluating large language models (LLMs) in the context of general intelligence. The researchers argue that human exams are a more suitable means of evaluating LLMs as they require a wider range of abilities such as language understanding, domain knowledge, and problem-solving skills. M3Exam is sourced from real and official human exam questions and exhibits three unique characteristics: multilingualism, multimodality, and a multilevel structure. The M3Exam dataset consists of 12,317 questions in 9 diverse languages with three educational levels. Approximately 23% of the questions require processing images for successful solving. The dataset includes context information, main question text, candidate options, correct answers, and meta information such as language, level, subject and image requirements. The selected languages in M3Exam span a wide range from high-resource languages like English and Chinese to extremely low-resource languages like Javanese. This diversity allows for comprehensive assessment of the multilingual capabilities of LLMs. The ratio of questions requiring images varies across countries. For the experiment setups various top-performing LLMs in either multilingual or multimodal settings are selected including ChatGPT (gpt-3.5-turbo), GPT-4 (gpt-4), Claude (Claude-instant), BLOOM (176B) and Vicuna (13B). Some models are open source while others are obtained via API calls. The performance of these LLMs on M3Exam is evaluated which reveals that current models still struggle with multilingual text particularly in low resource and non Latin script languages while multimodal LLMs also perform poorly with complex multimodal questions. Overall this study presents M3Exam as a benchmark for evaluating LLMs' general intelligence addressing the limitations of task specific benchmarks by incorporating multilingualism multimodality and a multilevel structure. The data and evaluation code are available on GitHub making it possible to track the development of LLMs over time using this valuable resource for comprehensively assessing their capabilities.

• M3Exam is a benchmark for evaluating large language models (LLMs) in the context of general intelligence.
• Human exams are considered more suitable for evaluating LLMs as they require a wider range of abilities.
• M3Exam dataset consists of 12,317 questions in 9 diverse languages with three educational levels.
• Approximately 23% of the questions require processing images for successful solving.
• The dataset includes context information, main question text, candidate options, correct answers, and meta information such as language, level, subject, and image requirements.
• The selected languages in M3Exam range from high-resource to extremely low-resource languages.
• Various top-performing LLMs in multilingual or multimodal settings are evaluated on M3Exam.
• Current models struggle with multilingual text, especially in low resource and non-Latin script languages.
• Multimodal LLMs perform poorly with complex multimodal questions.
• M3Exam addresses the limitations of task-specific benchmarks by incorporating multilingualism, multimodality, and a multilevel structure.

Summary- M3Exam is a test for big language models to see how smart they are. - People think human tests are better because they test more skills. - M3Exam has lots of questions in different languages and difficulty levels. - Some questions need to use pictures to find the answer. - The test has different kinds of information like the question, choices, and language. Definitions- Benchmark: A standard or test used to compare or measure something. - Language model: A computer program that understands and uses language. - Dataset: A collection of data or information used for studying or testing. - Processing: Doing something with information or data to get an answer or result. - Multilingual: Being able to understand and use more than one language.

Introducing M3Exam: A Novel Benchmark for Evaluating Large Language Models

In recent years, large language models (LLMs) have become increasingly popular due to their ability to generate human-like text. However, evaluating the general intelligence of LLMs is still a challenge as existing benchmarks are task specific and do not fully capture the range of abilities required by humans. To address this limitation, researchers from Carnegie Mellon University have developed M3Exam – a novel benchmark for assessing LLM’s general intelligence.

What is M3Exam?

M3Exam is sourced from real and official human exam questions and exhibits three unique characteristics: multilingualism, multimodality, and a multilevel structure. The dataset consists of 12,317 questions in 9 diverse languages with three educational levels ranging from primary school to university level. Approximately 23% of the questions require processing images for successful solving. The selected languages span a wide range from high-resource languages like English and Chinese to extremely low-resource languages like Javanese which allows for comprehensive assessment of the multilingual capabilities of LLMs.

Experiment Setup

For the experiment setups various top-performing LLMs in either multilingual or multimodal settings were selected including ChatGPT (gpt-3.5-turbo), GPT-4 (gpt-4), Claude (Claude-instant), BLOOM (176B) and Vicuna (13B). Some models were open source while others were obtained via API calls.

Results

The performance evaluation on M3Exam revealed that current models still struggle with multilingual text particularly in low resource and non Latin script languages while multimodal LLMs also perform poorly with complex multimodal questions. Overall this study presents M3Exam as an effective benchmark for comprehensively assessing the capabilities of LLMs when it comes to general intelligence tasks such as language understanding, domain knowledge, problem solving skills etc., which cannot be evaluated using task specific benchmarks alone. The data and evaluation code are available on GitHub making it possible to track the development of LLMs over time using this valuable resource .