# A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education

**Authors:** Jacob Doughty, Zipiao Wan, Anishka Bompelli, Jubahed Qayum, Taozhi Wang, Juran Zhang, Yujia Zheng, Aidan Doyle, Pragnya Sridhar, Arav Agarwal, Christopher Bogart, Eric Keylor, Can Kultur, Jaromir Savelka, Majd Sakr  
**arXiv:** [2312.03173](https://arxiv.org/abs/2312.03173)  
**Date:** 2023-12-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.03173v1/

## Key points

- Study explores use of large language models (LLMs), specifically GPT-4, for generating multiple-choice questions (MCQs) in Python programming courses
- LLM-powered system generated 651 MCQs compared to 449 human-crafted MCQs aligned to 246 learning objectives (LOs)
- GPT-4 capable of producing MCQs with clear language, single correct choice, and high-quality distractors
- Generated MCQs well-aligned with LOs compared to human-crafted ones
- Educators often focus more on aligning MCQs with module topics rather than LOs, resulting in misalignment
- LLM-generated MCQs showed better alignment with LOs compared to human-crafted ones
- Limitations include manual association of MCQs with LOs and evaluation setup used in the study
- Results suggest LLM-powered tools can generate high-quality MCQs while reducing workload for instructors
- This could make updating and revising assessments more efficient and allow instructors to focus on student engagement and curriculum enhancement.

## Summary

This study explores the use of large language models (LLMs), specifically GPT-4, for generating multiple-choice questions (MCQs) aligned with specific learning objectives (LOs) in Python programming courses. The researchers developed an LLM-powered system that generated 651 MCQs, which were then compared to 449 human-crafted MCQs aligned to 246 LOs from six Python courses. The findings indicate that GPT-4 was capable of producing MCQs with clear language, a single correct choice, and high-quality distractors. Additionally, the generated MCQs appeared to be well-aligned with the LOs. The study also highlights that educators often focus more on aligning MCQs with module topics rather than LOs, resulting in misalignment. However, the LLM-generated MCQs showed better alignment with LOs compared to human-crafted ones. While there are some limitations due to manual association of MCQs with LOs and the evaluation setup used in this study, the results suggest that LLM-powered tools can generate high-quality MCQs while reducing workload for instructors. This could potentially make updating and revising assessments more efficient and allow instructors to focus on student engagement and curriculum enhancement.