# Can Large Language Models Make the Grade? An Empirical Study Evaluating LLMs Ability to Mark Short Answer Questions in K-12 Education

**Authors:** Owen Henkel, Adam Boxer, Libby Hills, Bill Roberts  
**arXiv:** [2405.02985](https://arxiv.org/abs/2405.02985)  
**Date:** 2024-05-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.02985v1/

## Key points

- Researchers evaluated the effectiveness of Large Language Models (LLMs) in grading open text responses to short answer questions in K-12 education.
- The study focused on different combinations of GPT versions and prompt engineering strategies for marking real student answers in Science and History subjects across grade levels 5-16.
- Using a new dataset from Carousel, researchers found that GPT-4 with basic few-shot prompting achieved high performance (Kappa, 0.70), approaching human-level grading (0.75).
- Results suggest that LLMs could be valuable tools for supporting low-stakes formative assessment tasks in K-12 education, enhancing real-world education delivery.
- This research demonstrates the potential of LLMs in automating the grading process with high accuracy, opening up new possibilities for more efficient and effective educational assessment practices.

## Summary

In this empirical study, researchers evaluated the effectiveness of Large Language Models (LLMs) in grading open text responses to short answer questions in K-12 education. The study focused on assessing different combinations of GPT versions and prompt engineering strategies in marking real student answers across various subject areas (Science and History) and grade levels (ages 5-16). Using a new dataset from Carousel, a quizzing platform, the researchers found that GPT-4 with basic few-shot prompting achieved a high level of performance (Kappa, 0.70), closely approaching human-level grading (0.75). This research builds upon previous findings that GPT-4 can reliably score short answer reading comprehension questions at a level comparable to expert human raters. The results suggest that LLMs could serve as valuable tools for supporting low-stakes formative assessment tasks in K-12 education, offering important implications for enhancing real-world education delivery. Assessment and feedback are critical components of the learning process, with formative assessments playing a key role in improving learning outcomes. However, scaling formative assessment practices has traditionally been challenging due to costs and logistical demands. By demonstrating the potential of LLMs in automating the grading process with high accuracy, this study opens up new possibilities for more efficient and effective educational assessment practices.