UMass Study Reveals AI in Classroom Boosts Student Engagement Without Improving Test Scores

One class used AI, one didn't. Their exam scores were the same

As educators confront the rapid advancement of artificial intelligence and its role in the classroom, a semester-long experiment at the University of Massachusetts Amherst found that structured use of generative AI improved student engagement and confidence but did not raise exam scores. Christian Rojas, professor and chair in the Department of Resource Economics, led the controlled study in two back-to-back sections of an upper-division antitrust economics course with the same assignments, lectures and exams. One section of 29 students was encouraged to use AI tools such as ChatGPT with structured guidance and disclosure requirements, while a second section of 28 students was barred from using AI and received parallel non-AI study support. Rojas and co-authors Rong Rong, associate professor of resource economics, and Luke Bloomfield, senior lecturer of resource economics, found no measurable effect of AI on exam scores or final grades, but the AI-permitted class consistently reported higher satisfaction. Students with access to AI participated more in real-time classroom activities, concentrated their AI use into longer study sessions and developed more reflective learning habits, such as editing AI outputs, catching mistakes and preferring their own answers over the machine's. "It's not that AI helped students learn more -- it helped them learn more efficiently and confidently," says Rojas, who taught both sections of the course. "They spent less time outside the classroom on homework and exam preparation." Standardized course evaluations completed at the end of the semester were also significantly higher in the AI section, particularly in ratings of instructor preparation and use of class time. In addition, students with AI access were far more likely to say they intended to pursue careers that involve intensive use of AI. Rojas says the results should reassure educators that integrating AI into coursework can be accomplished without sacrificing academic rigor. He suggests a "permit with scaffolding" approach, where students are provided explicit instructions about effective uses of AI and clear disclosure requirements. "There's an opportunity for instructors to be more open about AI usage. Letting students engage with it just creates a different environment," Rojas notes. "It's been super impactful for me and the way I think about teaching." The experiment intentionally assigned the permission to use AI to an afternoon section, which historically performs slightly worse, making it a conservative test against detecting outsized AI effects. Students in both sections were assessed with paper-and-pencil exams, where notes, AI or other technology were not permitted. Even though the study spanned a considerable amount of time, Rojas cautions that many of the outcomes were self-reported by students and that the experiment involved a small sample size. The study is detailed in a new working paper, "Allowing Generative AI in Class: Evidence from a Semester-Long Controlled Teaching Study," which has been submitted for publication in a peer-reviewed journal.

This Professor Let Half His Class Use AI. Here’s What Happened

To use artificial intelligence, or not to use AI? That is the question that all students and their educators face in today’s age. While people have been endlessly arguing the pros and cons of incorporating tools like ChatGPT in the classroom, researchers have put it to the test with actual economics students, and the results are surprising. University of Massachusetts Amherst educators compared the impact of using AI versus not using AI in two back-to-back sections of an advanced antitrust economics course with the same lectures, assignments, and paper-and-pencil exams (without notes or technology). At the end of the semester, the experiment revealed that the structured use of generative AI increased student engagement and confidence, but not test results. “The core result is simple: permitting and scaffolding AI use did not raise proctored exam scores in this course. Yet the intervention meaningfully changed how students learned and how they felt about their learning,†the researchers wrote in a study published earlier this month in the journal Social Science Research Network. Students in one section were encouraged to use generative AI tools like ChatGPT with disclosure and guidance, while those in the other were not allowed to use AI and instead received parallel non-AI study guidance. Christian Rojas, co-author of the study and the professor who taught both sections, and his colleagues found that the students who could use AI exhibited better class participation. Furthermore, “by semester’s end, the two sections report similar frequency of AI use in their other courses, but students in the AI section concentrate that use into longer, more substantive sessions (15â€"30 minutes),†the researchers added. They also associated AI access with “more positive perceptionsâ€"especially regarding efficiency, confidence, and engagementâ€"and stronger intentions to continue using and studying AI, as well as choosing AI-intensive careers." Students with AI access matured more habits associated with reflective learning (learning from previous experiences or ideas), like editing AI-generated texts, identifying mistakes, and choosing their own answers over the AI’s. Moreover, they gave remarkably higher standardized course evaluations, especially in regard to instructor preparation and use of class time. All that said, AI use didn’t seem to impact exam scores or final grades. “It’s not that AI helped students learn moreâ€"it helped them learn more efficiently and confidently,†Rojas explained in a university statement. “They spent less time outside the classroom on homework and exam preparation.†According to the professor, the experiment demonstrates that educators can incorporate AI without their students taking shortcuts, though he is careful to note that the study analyzed a small sample size and involved a significant amount of self-reporting, meaning further research should be conducted on a greater scale. Nonetheless, “taken together, structured AI access with guardrails appears, in our setting, to reshape how students learn and feel about learning, without raising exam scores,†the researchers concluded in the study.

Students who used AI were happier in class, UMass study finds

Neal J. Riley is a digital producer for CBS Boston. He has been with WBZ-TV since 2014. His work has appeared in The Boston Globe and The San Francisco Chronicle. Neal is a graduate of Boston University. A new University of Massachusetts-Amherst study found that college students who were allowed to use artificial intelligence for a course were happier and more engaged in class than those who couldn't, though it didn't necessarily lead to better grades. The study involved two economics classes that had the same assignments, lectures and exams. UMass said one class was encouraged to use AI tools like ChatGPT "with structured guidance and disclosure requirements," while the other was banned from using AI. There was "no measurable effect" on test scores and grades, the study found, but those who were allowed to use AI "consistently reported higher satisfaction." Those students participated more in class and even found they preferred their own answers over ones that come from AI. The study concludes that AI helped students achieve "'the same for less,' primarily by reallocating effort and deepening the thought process rather than by boosting test performance." "It's not that AI helped students learn more -- it helped them learn more efficiently and confidently," study leader and economic professor Christian Rojas said in a statement. Students in the class that used AI were also far more likely to say they might pursue a career with an intensive AI focus. Massachusetts K-12 schools are currently working to identify the benefits of students using AI. Rojas said AI in the classroom can work if students are given explicit instructions about how to use it and disclose how it helped them. "There's an opportunity for instructors to be more open about AI usage. Letting students engage with it just creates a different environment," he said. "It's been super impactful for me and the way I think about teaching."