AI passes the Turing Test as GPT-4.5 fools humans 73% of the time in landmark study

Advanced AI Passes the Turing Test for the First Time - Neuroscience News

Summary: A milestone cognitive science study unveiled the first definitive empirical evidence that modern artificial intelligence can pass the iconic Turing test. The randomized, controlled study rigorously applied the 1950 framework created by British mathematician Alan Turing to evaluate whether state-of-the-art large language models (LLMs) could imitate human conversation so convincingly that real people could not tell them apart. Researchers discovered that when equipped with specific "persona" prompts, advanced models like GPT-4.5 were judged to be human 73% of the time, significantly outperforming actual human participants and fundamentally altering our understanding of machine intelligence. A new University of California San Diego study unveils the first empirical evidence that a modern artificial intelligence system can pass the Turing test -- a major scientific benchmark that asks whether a machine can imitate human conversation so convincingly that people can't reliably tell it apart from a real person. In a series of experiments, people were often unable to tell the difference between humans and advanced large language models (LLMs). The study published in the Proceedings of the National Academy of Sciences is the first to rigorously test LLMs with the method that British mathematician and "father of computer science" Alan Turing created in 1950. It is also the first time anyone has found that models were judged to be human as often as actual humans using the Turing framework. "What we found is that if given the right prompts, advanced LLMs can exhibit the same tone, directness, humor and fallibility as humans," said the study's corresponding author Cameron Jones, who conducted the research while earning his doctorate in Cognitive Science from UC San Diego. "While we know LLMs can easily produce knowledge on nearly every topic, this test showed that it can also convincingly display social behavioral traits, which has major implications for how we think of AI." In the test, a participant chats simultaneously with two other parties -- one is a human and the other is an LLM -- and the human "interrogator" must decide which party is the human. Across randomized, controlled, experiments with two independent participant groups -- UC San Diego undergraduates and online participants -- interrogators held brief, text-based conversations and then made their judgments. In the experiments participants chatted with four different LLMs -- GPT-4.5 and LLaMa-3.1-405B as state-of-the-art models -- and the researchers also included older baseline models for comparison. Those models included GPT-4o and ELIZA, a classic 1960s rules-based chatbot. Across the four LLMs, GPT-4.5 was judged to be the human 73% of the time, meaning interrogators selected it as "human" significantly more often than they selected the real human participant. LLaMa-3.1-405B, given the same prompt, was judged human 56% of the time -- statistically indistinguishable from the humans it was compared against. Baseline systems performed far worse: ELIZA and GPT-4o were selected as human only about 23% and 21% of the time overall. The study also reveals that scientists need to start thinking of the Turing Test differently, according to the authors. "The Turing test started as a way to ask whether machines could rival human intelligence," said study coauthor Ben Bergen, a professor of cognitive science at UC San Diego. "But now we know AI can answer many questions faster and more accurately than people can, so the real issue isn't raw brainpower. Seeing that machines can pass the test -- and seeing how they pass it -- forces us to rethink what it measures. Increasingly, it's measuring humanlikeness." Each LLM was given a "persona" prompt, asking it to adopt a specific human character and communication style. Bergen explained that the LLMs were not winning through shows of force of knowledge, they were winning because they made mistakes like a human would. "These traits aren't the kinds of math and logic problem-solving intelligence that I think Turing was imagining." Without explicit instructions, the models were far less likely to be mistaken for human: GPT-4.5 fell to a 36% win rate and LLaMa-3.1 to 38%, while baseline systems ELIZA (23%) and GPT-4o (21%) were chosen as human even less often. The same systems that could pass as human when given detailed instructions on what kind of character they should play were unable to adopt such characteristics without that guidance -- suggesting that while the models can behave in convincingly human ways, they often need humans to tell them how. "They have the ability to appear human-like, but maybe not as much the ability to figure out what it would take to appear human-like," Bergen said. The results carry real-world implications for trust online -- especially because the models that pass as human do so over the course of extended five or 15-minute conversations. "It's relatively easy to prompt these models to be indistinguishable from humans," said Jones, who is now an assistant professor of Psychology at Stony Brook University. "We need to be more alert; when you interact with strangers online people should be much less confident that they know they're talking to a human rather than an LLM." He also points to darker risks. "The Turing test is a game about lying for the models," Jones said. "One of the implications is that models seem to be really good at that." Bergen added that being unable to discern whether you're interacting with a human or bot can have serious consequences. "There are lots of people who would like to use bots to persuade people to share their social security numbers, and vote for their party, or buy their product," he said. The researchers note they hope the work sharpens public understanding of what these systems can now do -- and what kinds of safeguards society may need. To run the study, the researchers built an online interface designed to feel like a familiar messaging app. "For the interrogator, they have a split screen on their computer and they're asking questions to both witnesses," Jones said. "They know that one of those witnesses is a human and one of them is an AI." After five minutes -- and in a separate replication study, 15 minutes -- the interrogator had to decide which conversational partner was the real human. To confirm the results held beyond a single population, the researchers ran the study with two groups: UC San Diego undergraduates recruited through the SONA system and a broader online sample recruited through Prolific, a platform that pays participants to complete research studies. Nearly 500 people participated across the experiments. UC San Diego participants performed slightly better overall, possibly because they shared more "common ground" that could be used to probe one another, such as shared experiences and local campus details. A version of the Turing test interface used in the study is available at turingtest.live. Large Language Models Pass a Standard Three-Party Turing Test The Turing test has been widely discussed as a test of machine intelligence, but it also provides a measure of how humans distinguish other humans from machines. We evaluated 4 systems (ELIZA, GPT-4o, LLaMa-3.1-405B, and GPT-4.5) in two randomized, controlled, and preregistered Turing tests on independent populations. Participants had 5 min conversations simultaneously with another human participant and one of these systems before judging which conversational partner they thought was human. When prompted to adopt a humanlike persona, GPT-4.5 was judged to be the human 73% of the time: significantly more often than interrogators selected the real human participant. LLaMa-3.1, with the same prompt, was judged to be the human 56% of the time -- not significantly more or less often than the humans it was being compared to. Without these prompts, however, the same models performed significantly worse (38% and 36%), and did not consistently outperform baseline models, ELIZA and GPT-4o (23% and 21%, respectively). A third study replicated these results in 15-min games: two PERSONA-prompted models achieved pass rates of 56% and 59%. The results constitute empirical evidence that artificial systems can pass a standard three-party Turing test. Interrogators' reasoning focused more on stylistic and socio-emotional aspects of human behavior rather than more traditional notions of intelligence. The results have implications for debates about what kind of intelligence is exhibited by large language models, the social impacts these systems are likely to have, and the aspects of human behavior that people continue to see as unique.

AI Can Seem More Human Than Real Humans in a Classic Turing Test, Study Finds | Newswise

Newswise -- A new University of California San Diego study unveils the first empirical evidence that a modern artificial intelligence system can pass the Turing test -- a major scientific benchmark that asks whether a machine can imitate human conversation so convincingly that people can't reliably tell it apart from a real person. In a series of experiments, people were often unable to tell the difference between humans and advanced large language models (LLMs). The study published in the Proceedings of the National Academy of Sciences is the first to rigorously test LLMs with the method that British mathematician and "father of computer science" Alan Turing created in 1950. It is also the first time anyone has found that models were judged to be human as often as actual humans using the Turing framework. "What we found is that if given the right prompts, advanced LLMs can exhibit the same tone, directness, humor and fallibility as humans," said the study's corresponding author Cameron Jones, who conducted the research while earning his doctorate in Cognitive Science from UC San Diego. "While we know LLMs can easily produce knowledge on nearly every topic, this test showed that it can also convincingly display social behavioral traits, which has major implications for how we think of AI." In the test, a participant chats simultaneously with two other parties -- one is a human and the other is an LLM -- and the human "interrogator" must decide which party is the human. Across randomized, controlled, experiments with two independent participant groups -- UC San Diego undergraduates and online participants -- interrogators held brief, text-based conversations and then made their judgments. In the experiments participants chatted with four different LLMs -- GPT-4.5 and LLaMa-3.1-405B as state-of-the-art models -- and the researchers also included older baseline models for comparison. Those models included GPT-4o and ELIZA, a classic 1960s rules-based chatbot. Across the four LLMs, GPT-4.5 was judged to be the human 73% of the time, meaning interrogators selected it as "human" significantly more often than they selected the real human participant. LLaMa-3.1-405B, given the same prompt, was judged human 56% of the time -- statistically indistinguishable from the humans it was compared against. Baseline systems performed far worse: ELIZA and GPT-4o were selected as human only about 23% and 21% of the time overall. The study also reveals that scientists need to start thinking of the Turing Test differently, according to the authors. "The Turing test started as a way to ask whether machines could rival human intelligence," said study coauthor Ben Bergen, a professor of cognitive science at UC San Diego. "But now we know AI can answer many questions faster and more accurately than people can, so the real issue isn't raw brainpower. Seeing that machines can pass the test -- and seeing how they pass it -- forces us to rethink what it measures. Increasingly, it's measuring humanlikeness." Each LLM was given a "persona" prompt, asking it to adopt a specific human character and communication style. Bergen explained that the LLMs were not winning through shows of force of knowledge, they were winning because they made mistakes like a human would. "These traits aren't the kinds of math and logic problem-solving intelligence that I think Turing was imagining." Without explicit instructions, the models were far less likely to be mistaken for human: GPT-4.5 fell to a 36% win rate and LLaMa-3.1 to 38%, while baseline systems ELIZA (23%) and GPT-4o (21%) were chosen as human even less often. The same systems that could pass as human when given detailed instructions on what kind of character they should play were unable to adopt such characteristics without that guidance -- suggesting that while the models can behave in convincingly human ways, they often need humans to tell them how. "They have the ability to appear human-like, but maybe not as much the ability to figure out what it would take to appear human-like," Bergen said. The results carry real-world implications for trust online -- especially because the models that pass as human do so over the course of extended five or 15-minute conversations. "It's relatively easy to prompt these models to be indistinguishable from humans," said Jones, who is now an assistant professor of Psychology at Stony Brook University. "We need to be more alert; when you interact with strangers online people should be much less confident that they know they're talking to a human rather than an LLM." He also points to darker risks. "The Turing test is a game about lying for the models," Jones said. "One of the implications is that models seem to be really good at that." Bergen added that being unable to discern whether you're interacting with a human or bot can have serious consequences. "There are lots of people who would like to use bots to persuade people to share their social security numbers, and vote for their party, or buy their product," he said. The researchers note they hope the work sharpens public understanding of what these systems can now do -- and what kinds of safeguards society may need. To run the study, the researchers built an online interface designed to feel like a familiar messaging app. "For the interrogator, they have a split screen on their computer and they're asking questions to both witnesses," Jones said. "They know that one of those witnesses is a human and one of them is an AI." After five minutes -- and in a separate replication study, 15 minutes -- the interrogator had to decide which conversational partner was the real human. To confirm the results held beyond a single population, the researchers ran the study with two groups: UC San Diego undergraduates recruited through the SONA system and a broader online sample recruited through Prolific, a platform that pays participants to complete research studies. Nearly 500 people participated across the experiments. UC San Diego participants performed slightly better overall, possibly because they shared more "common ground" that could be used to probe one another, such as shared experiences and local campus details. A version of the Turing test interface used in the study is available at turingtest.live.