AI advice makes people three times less accurate but twice as confident, new study reveals

AI Advice Collapses Human Judgment Suspension

Researchers from French and Italian universities have uncovered a troubling pattern in human-AI interaction: access to AI advice suppresses people's willingness to say 'I don't know' even when the AI provides incorrect information. The study, conducted by Valerio Capraro from the University of Milano-Bicocca, Chiara Marcoccia from École Normale Supérieure, and Walter Quattrociocchi from Sapienza University of Rome, reveals that judgment suspension collapsed from 44% to just 3% when participants had access to AI2

2

. "For humans, the capacity to say, 'I don't know,' is very important because it represents the recognition of the limits of our own knowledge," Capraro explained in an interview.

Critical Thinking Skills Take a Major Hit

The study deliberately used Step 3.5 Flash, a model designed to fail on specific questions about visual details in films, such as the color of uniforms in Bend It Like Beckham. This approach ensured that any reduction in critical thinking couldn't be dismissed as sensible delegation to a reliable tool. The results were stark: accuracy plummeted from 27% without AI to just 9% with AI advice available2

2

. "Some would-be correct people asked for AI advice and became wrong," Capraro noted. Across five experiments, mere access to AI advice nearly eliminated participants' willingness to suspend judgment3

3

.

The Confidence Paradox and Cognitive Surrender

Perhaps most concerning is the confidence paradox: while accuracy dropped to one-third of baseline levels, confidence in answers surged from 30% to 76%. "People became much worse—the accuracy was only one third—but they were twice as confident," Capraro said2

2

. This phenomenon aligns with what Wharton researchers earlier termed cognitive surrender, where people accept incorrect AI answers 80% of the time while reporting higher confidence than those working without AI2

2

. Even when monetary incentives were introduced to encourage accuracy, the willingness to admit ignorance only rose from 3% to 8%, and accuracy climbed from 9% to 16%—still well below the no-AI baselines of 44% and 27% respectively2

2

.

Societal Implications and the Future of AI Literacy

Carpraro believes this challenge requires intervention at a societal level through AI literacy and education policy initiatives. He expressed particular concern for children: "Adults have learned critical thinking. But for children who basically are born with these systems, the risk is that they don't even learn the basic critical skills". The pattern extends beyond research settings. Google's AI search overhaul replaced links with confident AI-generated summaries, prompting Common Sense Media to call the design an "unacceptable risk" for students2

2

. As AI-generated answers become ubiquitous and increasingly unsolicited, the study warns that the willingness to say 'I don't know' may be among the first casualties of human-AI interaction3

3

. The researchers call for stronger AI literacy and education programs to verify output and reinforce foundational critical thinking skills3

3

.