The study conducted by Zhang et al. (2023) explores the use of artificial intelligence, specifically large language models (LLMs), for automating feedback in educational settings. The researchers focused on ChatGPT, a popular LLM, to provide feedback for Java programming assignments in a CS1 class. Their research aimed to address three key questions: 1) How do students perceive LLM-generated feedback in terms of its formative nature? 2) What are the comparative advantages of feedback prompts that include students' code versus those that do not? 3) What suggestions do students have for enhancing AI-generated feedback? To answer these questions, automated feedback was generated using the ChatGPT API for four lab assignments within the CS1 class. The results from surveys indicated that students viewed the AI-generated feedback as aligning well with established formative feedback guidelines. There was also a clear preference among students for feedback that incorporated their own code into the LLM prompt due to its specificity, clarity, and corrective nature. Additionally, the study revealed that students generally expected specific and corrective feedback with ample code examples but had varying opinions on the tone of the feedback provided by ChatGPT. Overall, the findings demonstrated that ChatGPT could effectively generate Java programming assignment feedback perceived as formative by students. This research not only showcases the potential of AI-driven tools like ChatGPT in educational settings but also sheds light on specific improvements necessary to enhance the utility of AI-generated feedback for student learning outcomes. By exploring students' perceptions and preferences regarding generative artificial intelligence feedback for programming assignments, this study contributes valuable insights to further optimize AI applications in educational contexts.
SummaryResearchers used a big computer program called ChatGPT to help teachers give feedback to students. They wanted to know if students liked the feedback and how it could be better. Students in a computer class got feedback from ChatGPT on their work. Most students thought the feedback was helpful, especially when it included their own work. The study showed that using ChatGPT can be good for helping students learn.
Definitions
- Large language models (LLMs): Big computer programs that can understand and generate human-like text.
- Automating: Making something work automatically without needing people to do it.
- Feedback: Information given to someone about their performance or work to help them improve.
- Formative: Feedback given during the learning process to guide improvement.
- AI-generated: Created by artificial intelligence, like a smart computer program.
- Java programming: A type of coding language used for creating software applications.
Introduction:
In recent years, the use of artificial intelligence (AI) has become increasingly prevalent in various fields, including education. AI-driven tools have shown great potential in automating tasks and providing personalized learning experiences for students. One area where AI has been gaining traction is in providing feedback on assignments and assessments. Traditional methods of feedback, such as manual grading by instructors or peer review, can be time-consuming and may not always provide timely and detailed feedback to students. This is where large language models (LLMs) come into play.
The study conducted by Zhang et al. (2023) explores the use of LLMs for automating feedback in educational settings, specifically for Java programming assignments in a CS1 class. The researchers focused on ChatGPT, a popular LLM developed by OpenAI, to generate feedback prompts for four lab assignments within the CS1 class. Their research aimed to address three key questions: 1) How do students perceive LLM-generated feedback in terms of its formative nature? 2) What are the comparative advantages of feedback prompts that include students' code versus those that do not? 3) What suggestions do students have for enhancing AI-generated feedback?
Methodology:
To answer these questions, automated feedback was generated using the ChatGPT API for four lab assignments within the CS1 class. A total of 100 undergraduate students participated in this study and were randomly assigned to two groups: one receiving traditional instructor-provided feedback and another receiving AI-generated feedback via ChatGPT.
After completing each assignment, participants were asked to complete a survey regarding their perceptions of the AI-generated feedback they received. The survey included questions about their overall satisfaction with the prompt's formative nature, preference between prompts with or without their own code incorporated into it, expectations from AI-generated feedback, and suggestions for improvement.
Results:
The results from surveys indicated that students viewed the AI-generated feedback as aligning well with established formative feedback guidelines. This suggests that ChatGPT can effectively generate feedback perceived as formative by students, which is crucial for their learning and development.
Furthermore, there was a clear preference among students for feedback that incorporated their own code into the LLM prompt. Students found this type of feedback to be more specific, clear, and corrective in nature compared to prompts without their code. This highlights the importance of incorporating students' work into AI-generated feedback to provide personalized and relevant suggestions for improvement.
The study also revealed that students generally expected specific and corrective feedback with ample code examples from AI-generated prompts. However, they had varying opinions on the tone of the feedback provided by ChatGPT. Some students found it too critical or harsh while others appreciated its directness.
Implications:
This research not only showcases the potential of AI-driven tools like ChatGPT in educational settings but also sheds light on specific improvements necessary to enhance the utility of AI-generated feedback for student learning outcomes. By exploring students' perceptions and preferences regarding generative artificial intelligence feedback for programming assignments, this study contributes valuable insights to further optimize AI applications in educational contexts.
One key implication is that incorporating students' own work into AI-generated prompts can significantly improve its effectiveness and usefulness for student learning outcomes. This not only provides personalized and relevant suggestions but also helps build a stronger connection between the student's work and the generated prompt.
Moreover, understanding student expectations from AI-generated feedback can help developers fine-tune these tools to better meet those expectations. For example, providing more positive reinforcement or adjusting the tone of the generated prompt based on individual preferences could lead to a more positive experience for students using these tools.
Conclusion:
In conclusion, Zhang et al.'s (2023) study provides valuable insights into how LLMs like ChatGPT can be used in educational settings to automate tasks such as providing timely and detailed feedback on assignments. The results demonstrate that AI-generated feedback can align well with established formative feedback guidelines and be perceived as formative by students. The study also highlights the importance of incorporating students' work into AI-generated prompts and understanding their expectations to further optimize these tools for educational contexts. As AI continues to advance, it is crucial to continue exploring its potential in education and finding ways to enhance its effectiveness for student learning outcomes.
The rapid evolution of artificial intelligence (AI), specifically large language models (LLMs), has opened opportunities for various educational applications. This paper explored the feasibility of utilizing ChatGPT, one of the most popular LLMs, for automating feedback for Java programming assignments in an introductory computer science (CS1) class. Specifically, this study focused on three questions: 1) To what extent do students view LLM-generated feedback as formative? 2) How do students see the comparative affordances of feedback prompts that include their code, vs. those that exclude it? 3) What enhancements do students suggest for improving AI-generated feedback? To address these questions, we generated automated feedback using the ChatGPT API for four lab assignments in the CS1 class. The survey results revealed that students perceived the feedback as aligning well with formative feedback guidelines established by Shute. Additionally, students showed a clear preference for feedback generated by including the students' code as part of the LLM prompt, and our thematic study indicated that the preference was mainly attributed to the specificity, clarity, and corrective nature of the feedback. Moreover, this study found that students generally expected specific and corrective feedback with sufficient code examples, but had diverged opinions on the tone of the feedback. This study demonstrated that ChatGPT could generate Java programming assignment feedback that students perceived as formative. It also offered insights into the specific improvements that would make the ChatGPT-generated feedback useful for students.
