# Assessing AI Detectors in Identifying AI-Generated Code: Implications for Education

**Authors:** Wei Hung Pan, Ming Jie Chok, Jonathan Leong Shan Wong, Yung Xin Shin, Yeong Shian Poon, Zhou Yang, Chun Yong Chong, David Lo, Mei Kuan Lim  
**arXiv:** [2401.03676](https://arxiv.org/abs/2401.03676)  
**Date:** 2024-01-08  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2401.03676v1/

## Key points

- Authors address the concern of educators regarding the use of Large Language Models (LLMs) like ChatGPT in programming education
- Focus on potential exploitation of imperfections in Artificial Intelligence Generated Content (AIGC) Detectors for academic misconduct
- Conducted an empirical study to examine how well AIGC Detectors can detect code generated by LLMs
- Generated code using different variants and collected a dataset consisting of 5,069 samples
- Created 13 sets of code problem variant prompts to instruct ChatGPT to generate outputs
- Assessed the performance of five AIGC detectors in distinguishing between human-written code and AI-generated code
- Existing AIGC Detectors perform poorly in accurately identifying AI-generated code compared to human-written code
- Highlights the need for further improvements in AIGC detection technology
- Emphasizes the challenges posed by LLMs in programming education
- Importance of developing robust AIGC Detectors to maintain academic integrity.

## Summary

In their paper "Assessing AI Detectors in Identifying AI-Generated Code: Implications for Education," authors Wei Hung Pan, Ming Jie Chok, Jonathan Leong Shan Wong, Yung Xin Shin, Yeong Shian Poon, Zhou Yang, Chun Yong Chong, David Lo and Mei Kuan Lim address the growing concern among educators regarding the use of Large Language Models (LLMs) like ChatGPT in programming education. The authors specifically focus on the potential exploitation of imperfections in Artificial Intelligence Generated Content (AIGC) Detectors for academic misconduct. To investigate this issue, they conduct an empirical study where they examine how well AIGC Detectors can detect code generated by LLMs. To gather data for their study, the authors generate code using different variants in response to coding problems and collect a dataset consisting of 5,069 samples. Each sample includes a textual description of a coding problem and its corresponding human-written Python solution code from various sources such as Quescol, Kaggle and LeetCode. Using this dataset, they create 13 sets of code problem variant prompts to instruct ChatGPT to generate outputs. They then assess the performance of five AIGC detectors in distinguishing between human-written code and AI-generated code. The results reveal that existing AIGC Detectors perform poorly in accurately identifying AI-generated code compared to human-written code. This finding highlights the need for further improvements in AIGC detection technology to ensure its effectiveness in detecting academic misconduct involving LLMs like ChatGPT. Overall,this research sheds light on the challenges posed by LLMs in programming education and emphasizes the importance of developing robust AIGC Detectors to maintain academic integrity. AI Detectors; Identifying AI-Generated Code; Education; Academic Misconduct; Large Language Models.