University Experiment Reveals AI's Impact on Exam Integrity: ChatGPT Outperforms Students

University Experiment Reveals AI's Impact on Exam Integrity: ChatGPT Outperforms Students

Peter Scarfe, a researcher at the University of Reading’s School of Psychology and Clinical Language Sciences, and his team conducted a controlled experiment to test the university's examination system against AI cheating. They created over 30 fake psychology student accounts and submitted answers produced by ChatGPT-4 to examination questions. The results were revealing: AI-generated work went largely undetected, with ChatGPT scoring better on average than human students.

The team used AI-generated submissions in five undergraduate modules, covering all three years of a psychology bachelor’s degree. Assignments included 200-word answers to short questions and 1,500-word essays. The markers were unaware of the experiment, and the AI-generated submissions were made without any editing, apart from minimal formatting for essays. Of the 63 AI-generated submissions, 94 percent went undetected, and nearly 84 percent received better grades than randomly selected students who took the same exams.

The experiment highlighted that most AI submissions detected were flagged not for being repetitive or robotic but for being exceptionally good. This led to a significant question: How should universities address AI-generated work?

Scarfe’s team researched AI-generated content detection tools like OpenAI’s GPTZero and Turnitin's systems. However, these tools performed well in lab settings but poorly in real-world applications. For example, GPTZero could flag AI-generated text 26 percent of the time with a 9 percent false positive rate, while Turnitin’s system, though advertised as highly effective, performed worse in practice. 

Adding to the challenge is the continuous improvement of large language models. The experiment conducted in the summer of 2023 used an older version of GPT-4, which has since seen multiple updates. This ongoing development suggests a race between AI content generation and detection tools, with detection systems currently lagging.

Moreover, AI systems designed to humanize AI-generated writing further complicate detection efforts. Scarfe noted that despite the challenges, there was one module where AI-generated work did not outperform human students: the final-year module requiring advanced critical thinking and analysis. However, he acknowledged that as AI improves, it will likely perform better on advanced assignments in the future.

Scarfe argued for the integration of AI into the education system, suggesting that universities should prepare students to use AI tools effectively in their professional careers. He compared this to the acceptance of spellcheckers and calculators, emphasizing that knowing how to use AI tools properly will be crucial.

Scarfe concluded that while integrating AI into education is inevitable, finding the best approach remains uncertain. He highlighted the importance of understanding and working with AI rather than relying on it entirely, drawing from his experience as a programmer where AI-assisted code writing still required human oversight and correction.