The Turing Test Challenged: GPT-4's Performance Sparks Debate on AI Intelligence

The Turing Test has a problem - and OpenAI's GPT-4.5 just exposed it

Most people know that the famous Turing Test, a thought experiment conceived by computer pioneer Alan Turing, is a popular measure of progress in artificial intelligence. Many mistakenly assume, however, that it is proof that machines are actually thinking. The latest research on the Turing Test from scholars at the University of California at San Diego shows that OpenAI's latest large language model, GPT-4.5, can fool humans into thinking that the AI model is a person in text chats, even more than a human can convince another person that they are human. Also: How to use ChatGPT: A beginner's guide to the most popular AI chatbot That's a breakthrough in the ability of gen AI to produce compelling output in response to a prompt. But even the researchers recognize that beating the Turing Test doesn't necessarily mean that "artificial general intelligence," or AGI, has been achieved -- a level of computer processing equivalent to human thought. The AI scholar Melanie Mitchell, a professor at the Santa Fe Institute in Santa Fe, New Mexico, has written in the scholarly journal Science that the Turing Test is less a test of intelligence per se and more a test of human assumptions. Despite high scores on the test, "the ability to sound fluent in natural language, like playing chess, is not conclusive proof of general intelligence," wrote Mitchell. The latest convincing-sounding achievement is described by Cameron Jones and Benjamin Bergen of UC San Diego in a paper published on the arXiv pre-print server this week, titled "Large Language Models Pass the Turing Test." Also: OpenAI expands GPT-4.5 rollout. Here's how to access (and what it can do for you) The paper is the latest installment in an experiment that Jones and Bergen have been running for years with participation from UC San Diego undergrads from the department. As the authors note, there have been decades of work on the problem. To date, there have been "more than 800 separate claims and counter-arguments having been made" about computers passing the test. The Turing Test was classically conceived by Turing as a round of passing text messages between a human "judge" and two "witnesses," one a human and one a computer. The computer and human witnesses were charged with convincing the human judge that they were human by the messages each sent. The judge knows only one of the two is human, but not which is which, and has to guess. That three-way form is essential. It means that if the judge mistakenly deems a computer human, then the judge was also missing the clues they should have gotten from the human about humanness. Also: With AI models clobbering every benchmark, it's time for human evaluation In other words, it's a test as much about how humans perceive and believe as it is about computer functioning. Jones and Bergen's updated design is posted online, and you can play the game yourself, as either judge or witness. As in prior versions, the human students at UC San Diego were asked to engage in a text chat app. Some of the students played the judge and some the human witness in randomized five-minute chat exchanges, while GPT-4.5 played the computer witness. Each judge had simultaneous conversations with both a human and a computer witness to fit the three-way form of Turing's formulation. To play the computer witness, multiple large language models were used, input with the judge's challenges in a prompt. GPT-4.5 was tested with two different prompts. One prompt was given minimal information. But a second, called PERSONA, was given a lot of extra details in the prompt, including "additional instructions on what kind of persona to adopt in responding to the interrogator: specifically a young person who is introverted, knowledgeable about internet culture, and uses slang," the authors write. Three other AI programs were tested: OpenAI's previous model, GPT-4o; Meta Platforms's Llama 3.1 405b; and a very ancient chat bot program from the 1960s, called ELIZA. ELIZA was included because, as a more primitive program, it could act as a check on whether the large language models fooled a human simply by random chance or because they were actually better programs. GPT-4.5, report Jones and Bergen, had a "win rate" of 73%, meaning it fooled the human judge into declaring it human 73% of the time. By contrast, the version of GPT-4.5 with the minimalist prompt and the three other AI programs did nowhere near as well, with scores ranging from 21% to 56%, even when the other language models were given a similarly rich PERSONA prompt. Also: Are ChatGPT Plus or Pro worth it? Here's how they compare to the free version Moreover, GPT-4.5 was way above a test of GPT-4 last year by Jones and Bergen. It had a win rate of only 54%, just above random chance. Jones and Bergen conclude that "interrogators were not only unable to identify the real human witness, but were in fact more likely to believe this model was human than that other human participants were." So, you might ask what it means that humans do a poor job of telling a computer and a person apart based on chat messages. The "most controversial question" about Turing's problem over the decades is whether it is actually measuring intelligence, Jones and Bergen acknowledge. One way of looking at it is that machines have gotten so good that they can "adapt their behaviour to different scenarios that makes them so flexible: and apparently so capable of passing as human," they observe. The PERSONA prompt, created by humans, is something to which GPT-4.5 "adapted" itself to win. Again, it's a genuine technical breakthrough in the AI model's capabilities. Also: ChatGPT Plus is free for students now - how to grab this deal before finals However, a big complaint with the test is that humans might simply be bad at recognizing intelligence. The authors conclude their experiment is evidence of that, at least partially. They note that 23% of the time, the older ELIZA program fooled human judges. As they relate, that was not because it was somehow obviously smarter. "Many participants selected ELIZA because it did not meet their expectations of an AI system (e.g. 'they were sarcastic' or 'I don't think AI would be so rude'.)," they write. Those guesses, they write, "suggest that interrogators' decisions incorporate complex assumptions about how humans and AI systems might be likely to behave in these contexts, beyond simply selecting the most intelligent-seeming agent." In fact, the human judges didn't ask very much about knowledge in their challenges, even though Turing thought that would be the main criterion. "[O]ne of the reasons most predictive of accurate verdicts" by the human judge, they write, "was that a witness was human because they lacked knowledge." All this means humans were picking up on things such as sociability rather than intelligence, leading Jones and Bergen to conclude that "Fundamentally, the Turing test is not a direct test of intelligence, but a test of humanlikeness." For Turing, intelligence may have appeared to be the biggest barrier for appearing humanlike, and hence to passing the Turing test. But as machines become more similar to us, other contrasts have fallen into sharper relief, to the point where intelligence alone is not sufficient to appear convincingly human. Left unsaid by the authors is that humans have become so used to typing into a computer -- to a person or to a machine -- that the Test is no longer a novel test of human-computer interaction. It's a test of online human habits. One implication is that the test needs to be expanded. The authors write that "intelligence is complex and multifaceted," and "no single test of intelligence could be decisive." Also: Gemini Pro 2.5 is a stunningly capable coding assistant - and a big threat to ChatGPT In fact, they suggest the test could come out very different with different designs. Experts in AI, they note, could be tested as a judge cohort. They might judge differently than lay people because they have different expectations of a machine. If a financial incentive were added to raise the stakes, human judges might scrutinize more closely and more thoughtfully. Those are indications that attitude and expectations play a part. "To the extent that the Turing test does index intelligence, it ought to be considered among other kinds of evidence," they conclude. That suggestion seems to square with an increasing trend in the AI research field to involve humans "in the loop," assessing and evaluating what machines do. Left open is the question of whether human judgment will ultimately be enough. In the movie Blade Runner, the "replicant" robots in their midst have gotten so good that humans rely on a machine, "Voight-Kampff," to detect who's human and who's robot. As the quest goes on to reach AGI, and humans realize just how difficult it is to say what AGI is or how they would recognize it if they stumbled upon it, perhaps humans will have to rely on machines to assess machine intelligence. Also: 10 key reasons AI went mainstream overnight - and what happens next Or, at the very least, they may have to ask machines what machines "think" about humans writing prompts to try to make a machine fool other humans.

The Rise of Fluid Intelligence

Deep down, Sam Altman and François Chollet share the same dream. They want to build AI models that achieve "artificial general intelligence," or AGI -- matching or exceeding the capabilities of the human mind. The difference between these two men is that Altman has suggested that his company, OpenAI, has practically built the technology already. Chollet, a French computer scientist and one of the industry's sharpest skeptics, has said that notion is "absolutely clown shoes." When I spoke with him earlier this year, Chollet told me that AI companies have long been "intellectually lazy" in suggesting that their machines are on the path to a kind of supreme knowledge. At this point, those claims are based largely on the programs' ability to pass specific tests (such as the LSAT, Advanced Placement Biology, and even an introductory sommelier exam). Chatbots may be impressive. But in Chollet's reckoning, they're not genuinely intelligent. Chollet, like Altman and other tech barons, envisions AI models that can solve any problem imaginable: disease, climate change, poverty, interstellar travel. A bot needn't be remotely "intelligent" to do your job. But for the technology to fulfill even a fraction of the industry's aspirations -- to become a researcher "akin to Einstein," as Chollet put it to me -- AI models must move beyond imitating basic tasks, or even assembling complex research reports, and display some ingenuity. Chollet isn't just a critic, nor is he an uncompromising one. He has substantial experience with AI development and created a now-prominent test to gauge whether machines can do this type of thinking. For years, he has contributed major research to the field of deep learning, including at Google, where he worked as a software engineer from 2015 until this past November; he wants generative AI to be revolutionary, but worries that the industry has strayed. In 2019, Chollet created the Abstraction and Reasoning Corpus for Artificial General Intelligence, or ARC-AGI -- an exam designed to show the gulf between AI models' memorized answers and the "fluid intelligence" that people have. Drawing from cognitive science, Chollet described such intelligence as the ability to quickly acquire skills and solve unfamiliar problems from first principles, rather than just memorizing enormous amounts of training data and regurgitating information. (Last year, he launched the ARC Prize, a competition to beat his benchmark with a $1 million prize fund.) You, a human, would likely pass this exam. But for years, chatbots had a miserable time with it. Most people, despite having never encountered ARC-AGI before, get scores of roughly 60 to 70 percent. GPT-3, the program that became ChatGPT, the legendary, reality-distorting bot, scored a zero. Only recently have the bots started to catch up. How could such powerful tools fail the test so spectacularly for so long? This is where Chollet's definition of intelligence comes in. To him, a chatbot that has analyzed zillions of SAT-style questions, legal briefs, and lines of code is not smart so much as well prepared -- for the SAT, a law-school exam, advanced coding problems, whatever. A child figuring out tricky word problems after just learning how to multiply and divide, meanwhile, is smart. ARC-AGI is simple, but it demands a keen sense of perception and, in some sense, judgment. It consists of a series of incomplete grids that the test-taker must color in based on the rules they deduce from a few examples; one might, for instance, see a sequence of images and observe that a blue tile is always surrounded by orange tiles, then complete the next picture accordingly. It's not so different from paint by numbers. The test has long seemed intractable to major AI companies. GPT-4, which OpenAI boasted in 2023 had "advanced reasoning capabilities," didn't do much better than the zero percent earned by its predecessor. A year later, GPT-4o, which the start-up marketed as displaying "text, reasoning, and coding intelligence," achieved only 5 percent. Gemini 1.5 and Claude 3.7, flagship models from Google and Anthropic, achieved 5 and 14 percent, respectively. These models may have gotten lucky on a few puzzles, but to Chollet they hadn't evinced a shred of abstract reasoning. "If you were not intelligent, like the entire GPT series," he told me, "you would score basically zero." In his view, the tech barons were not even on the right path to building their artificial Einstein. Read: The GPT era is already ending Chollet designed the grids to be highly distinctive, so that similar puzzles or relevant information couldn't inadvertently be included in a model's training data -- a common problem with AI benchmarks. A test taker must start anew with each puzzle, applying basic notions of counting and geometry. Most other AI evaluations and standardized tests are crude by comparison -- they aren't designed to evaluate a distinct, qualitative aspect of thinking. But ARC-AGI checks for the ability to "take concepts you know and apply them to new situations very efficiently," Melanie Mitchell, an AI researcher at the Santa Fe Institute, told me. To improve their performance, Silicon Valley needed to change its approach. Scaling AI -- building bigger models with more computing power and more training data -- clearly wasn't helping. OpenAI was first to market with a model that even came close to the right kind of problem-solving. The firm announced a so-called reasoning model, o1, this past fall that Altman later called "the smartest model in the world." Mark Chen, OpenAI's chief research officer, told me the program represented a "new paradigm." The model was designed to check and revise its approach to any question and to spend more time on harder ones, as a human might. An early version of o1 scored 18 percent on ARC-AGI -- a definite improvement, but still well below human performance. A later iteration of o1 hit 32 percent. OpenAI was still "a long way off" from fluid intelligence, Chollet told me in September. That was about to change. In late December, OpenAI previewed a more advanced reasoning model, o3, that scored a shocking 87 percent on ARC-AGI -- making it the first AI to match human performance on the test and the best-performing model by far. Chollet described the program as a "genuine breakthrough." o3 appeared able to combine different strategies on the fly, precisely the kind of adaptation and experimentation needed to succeed on ARC-AGI. Unbeknownst to Chollet, OpenAI had kept track of his test "for quite a while," Chen told me in January. Chen praised the "genius of ARC," calling its resistance to memorized answers a good "way to test generalization, which we see as closely linked to reasoning." And as the start-up's reasoning models kept improving, ARC-AGI resurfaced as a meaningful challenge -- so much so that the ARC Prize team collaborated with OpenAI for o3's announcement, during which Altman congratulated them on "making such a great benchmark." Chollet, for his part, told me he feels "pretty vindicated." Major AI labs were adopting, even standardizing, his years-old ideas about fluid intelligence. It is not enough for AI models to memorize information: They must reason and adapt. Companies "say they have no interest in the benchmark, because they are bad at it," Chollet said. "The moment they're good at it, they will love it." Many AI proponents were quick to declare victory when o3 passed Chollet's test. "AGI has been achieved in 2024," one start-up founder wrote on X. Altman wrote in a blog post that "we are now confident we know how to build AGI as we have traditionally understood it." Since then, Google, Anthropic, xAI, and DeepSeek have launched their own "reasoning" models, and the CEO of Anthropic, Dario Amodei, has said that artificial general intelligence could arrive within a couple of years. But Chollet, ever the skeptic, wasn't sold. Sure, AGI might be getting closer, he told me -- but only in the sense that it had previously been "infinitely" far away. And just as this hurdle was cleared, he decided to raise another. Last week, the ARC Prize team released an updated test, called ARC-AGI-2, and it appears to have sent the AIs back to the drawing board. The full o3 model has not yet been tested, but a version of o1 dropped from 32 percent on the original puzzles to just 3 percent on the new version, and a "mini" version of o3 currently available to the public dropped from roughly 30 percent to below 2 percent. (An OpenAI spokesperson declined to say whether the company plans to run the benchmark with o3.) Other flagship models from OpenAI, Anthropic, and Google have achieved roughly 1 percent, if not lower. Human testers average about 60 percent. If ARC-AGI-1 was a binary test for whether a model had any fluid intelligence, Chollet told me last month, the second version aims to measure just how savvy an AI is. Chollet has been designing these new puzzles since 2022; they are, in essence, much harder versions of the originals. Many of the answers to ARC-AGI were immediately recognizable to humans, while on ARC-AGI-2, people took an average of five minutes to find the solution. Chollet believes the way to get better on ARC-AGI-2 is to be smarter, not to study harder -- a challenge that may help push the AI industry to new breakthroughs. He is turning the ARC Prize into a nonprofit dedicated to designing new benchmarks to guide the technology's progress, and is already working on ARC-AGI-3. Read: DOGE's plans to replace humans with AI are already under way Reasoning models take bizarre and inhuman approaches to solving these grids, and increased "thinking" time will come at substantial cost. To hit 87 percent on the original ARC-AGI test, o3 spent roughly 14 minutes per puzzle and, by my calculations, may have required hundreds of thousands of dollars in computing and electricity; the bot came up with more than 1,000 possible answers per grid before selecting a final submission. Mitchell, the AI researcher, said this approach suggests some degree of trial and error rather than efficient, abstract reasoning. Chollet views this inefficiency as a fatal flaw, but corporate AI labs do not. If chatbots achieve fluid intelligence in this way, it will not be because the technology approximates the human mind: You can't just stuff more brain cells into a person's skull, but you can give a chatbot more computer chips. In the meantime, OpenAI is "shifting towards evaluations that reflect utility as well," Chen told me, such as tests of an AI model's ability to navigate and take actions on the web -- which will help the company make better, although not necessarily smarter, products. OpenAI itself, not some third-party test, will ultimately decide when its products are useful, how to price them (perhaps $20,000 a year for a "Phd-level" bot, according to one report), and whether they've achieved AGI. Indeed, the company may already have its own key AGI metric, of a sort: As The Information reported late last year, Microsoft and OpenAI have come to an agreement defining AGI as software capable of generating roughly $100 billion in profits. According to documents OpenAI distributed to investors, that determination "is in the 'reasonable discretion' of the board of OpenAI." And there's the problem: Nobody agrees on what's being measured, or why. If AI programs are bad at Chollet's test, maybe it just means that they have a hard time visualizing colorful grids rather than anything deeper. And bots that never solve ARC-AGI-2 could generate $100 billion in profits some day. Any specific test -- the LSAT or ARC-AGI or a coding puzzle -- will inherently contradict the notion of general intelligence; the term's defining trait may be its undefinability. The deeper issue, perhaps, is that human intelligence is poorly understood, and gauging it is an infamously hard and prejudiced task. People have knacks for different things, or might arrive at the same result -- the answer to a math problem, the solution to an ARC-AGI grid -- via very different routes. A person who scores 30 percent on ARC-AGI-2 is in no sense inferior to someone who scores 90 percent. The collision of those differing routes and minds is what sparks debate, creativity, and beauty. Intentions, emotions, and lived experiences drive people as much as any logical reasoning. Human cognitive diversity, in other words, is a glorious jumble. How do you even begin to construct an artificial version of that? And when that diversity is already so abundant, do you really want to?

What is artificial general intelligence and how does it differ from other types of AI?

Turns out, training artificial intelligence systems is not unlike raising a child. That's why some AI researchers have begun mimicking the way children naturally acquire knowledge and learn about the world around them -- through exploration, curiosity, gradual learning, and positive reinforcement. "A lot of problems with AI algorithms today could be addressed by taking ideas from neuroscience and child development," says Christopher Kanan, an associate professor in the Department of Computer Science at the University of Rochester, and an expert in artificial intelligence, continual learning, vision, and brain-inspired algorithms. Of course, learning and being able to reason like a human -- just faster and possibly better -- opens up questions about how best to keep humans safe from ever-advancing AI systems. That's why Kanan says all AI systems need to have guardrails built in, but doing so at the very end of the development is too late. "It shouldn't be the last step, otherwise we can unleash a monster." What is artificial general intelligence and how does it differ from other types of AI? AI involves creating computer systems that can perform tasks that typically require human intelligence, such as perception, reasoning, decision-making, and problem-solving. Traditionally, much of AI research has focused on building systems designed for specific tasks -- so-called artificial narrow intelligence (ANI). Examples include systems for image recognition, voice assistants, or playing strategic games, all of which can perform their tasks exceptionally well, often surpassing humans. Then there is artificial general intelligence (AGI), which aims to build systems capable of understanding, reasoning, and learning across a wide range of tasks, much like humans do. Achieving AGI remains a major goal in AI research but has not yet been accomplished. Beyond AGI lies artificial superintelligence (ASI) -- a form of AI vastly exceeding human intelligence in virtually every domain, which remains speculative and is currently confined to science fiction. In my lab, we're particularly interested in moving closer to artificial general intelligence by drawing inspiration from neuroscience and child development, enabling AI systems to learn and adapt continually, much like human children do. What are some of the ways that AI can 'learn?' ANI is successful thanks to deep learning, which since about 2014 has been used to train these systems to learn from large amounts of data annotated by humans. Deep learning involves training large artificial neural networks composed of many interconnected layers. Today, deep learning underpins most modern AI applications, from computer vision and natural language processing to robotics and biomedical research. These systems excel at tasks like image recognition, language translation, playing complex games such as Go and chess, and generating text, images, and even code. A large language model (LLM) like OpenAI's GPT-4 is trained on enormous amounts of text using self-supervised learning. This means the model learns by predicting the next word or phrase from existing text, without explicit human guidance or labels. These models are typically trained on trillions of words -- essentially the entirety of human writing available online, including books, articles, and websites. To put this in perspective, if a human attempted to read all this text, it would take tens of thousands of lifetimes. Following this extensive initial training, the model undergoes supervised fine-tuning, where humans provide examples of preferred outputs, guiding the model toward generating responses that align closely with human preferences. Lastly, techniques such as reinforcement learning with human feedback (RLHF) are applied to shape the model's behavior by defining acceptable boundaries for what it can or cannot generate. What are AIs really good at? They are excellent at tasks involving human languages, including translation, essay writing, text editing, providing feedback, and acting as personalized writing tutors. They can pass standardized tests. For example, OpenAI's GPT-4 achieves top-tier scores on really challenging tests such as the Bar Exam (90th percentile), LSAT (88th percentile), GRE Quantitative (80th percentile), GRE Verbal (99th percentile), USMLE, and several Advanced Placement tests. They even excel on Ph.D.-level math exams. Surprisingly, studies have shown they have greater emotional intelligence than humans. Beyond tests, LLMs can serve as co-scientists, assisting researchers in generating novel hypotheses, drafting research proposals, and synthesizing complex scientific literature. They're increasingly being incorporated into multimodal systems designed for vision-language tasks, robotics, and real-world action planning. What are some of the current limitations of generative AI tools? LLMs can still "hallucinate," which means they confidently produce plausible-sounding but incorrect information. Their reasoning and planning capabilities, while rapidly improving, are still limited compared to human-level flexibility and depth. And they don't continually learn from experience; their knowledge is effectively frozen after training, meaning they lack awareness of recent developments or ongoing changes in the world. Current generative AI systems also lack metacognition, which means they typically don't know what they don't know, and they rarely ask clarifying questions when faced with uncertainty or ambiguous prompts. This absence of self-awareness limits their effectiveness in real-world interactions. Humans excel at continual learning, where early-acquired skills serve as the basis for increasingly complex abilities. For instance, infants must first master basic motor control before progressing to walking, running, or even gymnastics. Today's LLMs neither demonstrate nor are effectively evaluated on this type of cumulative, forward-transfer learning. Addressing this limitation is a primary goal of my lab's research. What main challenges and risks does AI pose? Generative AI is already significantly transforming the workplace. It's particularly disruptive for white-collar roles -- positions that traditionally require specialized education or expertise -- because AI copilots empower individual workers to substantially increase their productivity; they can transform novices into operating at a level closer to that of experts. This increased productivity means companies could operate effectively with significantly fewer employees, raising the possibility of large-scale reductions in white-collar roles across many industries. In contrast, jobs requiring human dexterity, creativity, leadership, and direct physical interaction, such as skilled trades, health care positions involving direct patient care, or craftsmanship, are unlikely to be replaced by AI anytime soon. While scenarios like Nick Bostrom's famous "Paperclip Maximizer," in which AGI inadvertently destroys humanity, are commonly discussed, I think the greater immediate risk are humans who may deliberately use advanced AI for catastrophic purposes. Efforts should focus on international cooperation, responsible development, and investment in academic safety AI research. To ensure AI is developed and used safely, we need regulation around specific applications. Interestingly, the people asking for government regulation now are the ones who run the AI companies. But personally, I'm also worried about regulation that could eliminate open-source AI efforts, stifle innovation, and concentrate the benefits of AI among the few. What are the chances of achieving artificial general intelligence (AGI)? The three "godfathers" of modern AI and Turing Award winners -- Yoshua Bengio, Geoffrey Hinton, and Yann LeCun -- all agree that achieving AGI is possible. Recently, Bengio and Hinton have expressed significant concern, cautioning that AGI could potentially pose an existential risk to humanity. Nevertheless, I don't think any of them -- or I -- believe that today's LLM architectures alone will be sufficient to achieve true AGI. LLMs inherently reason using language, whereas for humans, language primarily serves as a means of communication rather than a primary medium for thought itself. This reliance on language inherently constrains the ability of LLMs to engage in abstract reasoning or visualization, limiting their potential for broader, human-like intelligence.