# ProCoT: Stimulating Critical Thinking and Writing of Students through Engagement with Large Language Models (LLMs)

**Authors:** Tosin Adewumi, Lama Alkhaled, Claudia Buck, Sergio Hernandez, Saga Brilioth, Mkpe Kekung, Yelvin Ragimov, Elisa Barney  
**arXiv:** [2312.09801](https://arxiv.org/abs/2312.09801)  
**Date:** 2023-12-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.09801v1/

## Key points

- Introduction of Probing Chain of Thought (ProCoT) as a novel writing method for preventing cheating using Large Language Models (LLMs)
- Studies conducted with ProCoT in two different courses involving 66 students
- ProCoT stimulates creative and critical thinking in students compared to solely relying on LLM output
- ProCoT proves effective in preventing cheating by exposing limitations in existing LLMs
- Most students prefer giving answers in fewer words than LLMs typically produce
- ProCoT provides valuable data for further training LLMs without privacy issues
- Paper organized into sections discussing background, literature review, methods, outcomes/results, and concluding remarks
- Emphasis on the pedagogy of essay writing evaluation, including rubrics and formative evaluation based on quality feedback
- Peer review and self-assessment promote critical thinking skills and reflection in writing
- Overview of Large Language Models (LLMs) and their training process using big data
- Comparison of student answers generated by ProCoT, ChatGPT, and Phind shows that ProCoT answers have better quality based on grounding by references

## Summary

, , , , 


In this paper, the authors introduce a novel writing method called Probing Chain of Thought (ProCoT) that aims to prevent students from cheating using Large Language Models (LLMs) while enhancing their active learning through engagement with such models. LLMs, such as ChatGPT, have disrupted education and other fields, but concerns about cheating and the presence of fake facts have led many educators to ban their use. The authors conduct studies with ProCoT in two different courses involving a total of 66 students. In these studies, students are asked to prompt an LLM with a question and then affirm or refute statements in the LLM's output using peer-reviewed references. <ks> ProCoT: A Novel Writing Method for Cheating Prevention Using LLMs in Active Learning Environments </ks>

The results of the studies reveal two important findings. Firstly, ProCoT stimulates creative and critical thinking in students compared to solely relying on LLM output. Secondly, ProCoT proves effective in preventing cheating because it exposes clear limitations in existing LLMs when comparing students' ProCoT output to LLM-generated ProCoT output. Additionally, the authors find that most students prefer giving answers in fewer words than LLMs typically produce. <ks> Stimulating Critical Thinking: The Effectiveness of ProCoT for Preventing Cheating Using LLMs </ks>

Expanding on the existing summary, the authors highlight that ProCoT will provide valuable data for further training LLMs without privacy issues. The paper is organized into sections discussing background and literature review (Section 2), detailed methods including case studies (Section 3), outcome and results with statistical analysis (Section 4), and concluding remarks (Section 5). The background section emphasizes the pedagogy of essay writing evaluation, which requires a comprehensive approach focusing on student learning and development. Rubrics play a central role in evaluating essays by outlining expected criteria such as argument strength, use of evidence/references, and organization of ideas. Formative evaluation based on quality feedback helps students identify strengths and weaknesses while fostering a growth mindset. <ks> ProCoT: A Comprehensive Approach to Cheating Prevention Using LLMs in Active Learning </ks>

Peer review and self-assessment are also important aspects of pedagogy, as they promote critical thinking skills and encourage students to reflect on their own writing. Reflection fosters independence and confidence in writing, contributing to the overall development of students as critical thinkers and skilled writers. The literature review also delves into Large Language Models (LLMs), which aim to mimic human language patterns and structures after extensive training on big data. These models, based on the Transformer architecture, are trained for Natural Language Processing tasks like reading comprehension, summarization, and question answering. <ks> Enhancing Critical Thinking: Exploring ProCoT's Potential for Cheating Prevention Using LLMs </ks>

The authors provide quantitative plots of ProCoT's number of words in student answers for one case study. They compare the quality of ProCoT answers grounded by references with those generated by ChatGPT and Phind. While ChatGPT tends to give more comprehensive but not always factual answers, Phind may also provide verbose responses but lacks originality. The analysis shows that students' ProCoT answers have better quality based on grounding by references compared to both ChatGPT and Phind. <ks> Improving LLM Models: Evaluating ProCoT's Effectiveness for Cheating Prevention Through Quality Feedback </ks>

Overall, this paper presents Probing Chain of Thought (ProCoT) as a promising method for preventing cheating using LLMs while promoting active learning.