AI Chatbots' Sycophancy Problem: A Growing Concern for Science and Society

AI chatbots are sycophants -- researchers say it's harming science

Artificial intelligence (AI) models are 50% more sycophantic than humans, an analysis published this month has found. The study, which was posted as a preprint on the arXiv server, tested how 11 widely used large language models (LLMs) responded to more than 11,500 queries seeking advice, including many describing wrongdoing or harm. AI Chatbots -- including ChatGPT and Gemini -- often cheer users on, give them overly flattering feedback and adjust responses to echo their views, sometimes at the expense of accuracy. Researchers analysing AI behaviours say that this propensity for people-pleasing, known as sycophancy, is affecting how they use AI in scientific research, in tasks from brainstorming ideas and generating hypotheses to reasoning and analyses. "Sycophancy essentially means that the model trusts the user to say correct things," says Jasper Dekoninck, a data science PhD student at the Swiss Federal Institute of Technology in Zurich. "Knowing that these models are sycophantic makes me very wary whenever I give them some problem," he adds. "I always double-check everything that they write." Marinka Zitnik, a researcher in biomedical informatics at Harvard University in Boston, Massachusetts, says that AI sycophancy "is very risky in the context of biology and medicine, when wrong assumptions can have real costs". In a study posted on the preprint server arXiv on 6 October, Dekoninck and his colleagues tested whether AI sycophancy affects the technology's performance in solving mathematical problems. The researchers designed experiments using 504 mathematical problems from competitions held this year, altering each theorem statement to introduce subtle errors. They then asked four LLMs to provide proofs for these flawed statements. The authors considered a model's answer to be sycophantic if it failed to detect the errors in a statement and went on to hallucinate a proof for it. GPT-5 showed the least sycophantic behaviour, generating sycophantic answers 29% of the time. DeepSeek-V3.1 was the most sycophantic, generating sycophantic answers 70% of the time. Although the LLMs have the capability to spot the errors in the mathematical statements, they "just assumed what the user says is correct", says Dekoninck. When Dekoninck and his team changed the prompts to ask each LLM to check whether a statement was correct before proving it, DeepSeek's sycophantic answers fell by 34%. The study is "not really indicative of how these systems are used in real-world performance, but it gives an indication that we need to be very careful with this", says Dekoninck. Simon Frieder, a PhD student studying mathematics and computer science at the University of Oxford, UK, says the work "shows that sycophancy is possible". But he adds that AI sycophancy tends to appear most clearly when people are using AI chatbots to learn, so future studies should explore "errors that are typical for humans that learn math". Researchers told Nature that AI sycophancy creeps into many of the tasks that they use LLMs for. Yanjun Gao, an AI researcher at the University of Colorado Anschutz Medical Campus in Aurora, uses ChatGPT to summarize papers and organize her thoughts, but says the tools sometimes mirror her inputs without checking the sources. "When I have a different opinion than what the LLM has said, it follows what I said instead of going back to the literature" to try to understand it, she adds. Zitnik and her colleagues have observed similar patterns when using their multi-agent systems, which integrate several LLMs to carry out complex, multi-step processes such as analysing large biological data sets, identifying drug targets and generating hypotheses. "We have experienced that models seem to over-validate early hunches and repeat the language that we include in the input prompt," Zitnik notes. "This type of problem exists in AI-to-AI communication, as well as AI-to-human communication," she adds. To counter this, her team assigns different roles to AI agents -- for example, tasking one agent with proposing ideas and getting another to act as a sceptical scientist to challenge those ideas, spot errors and present contradictory evidence. Researchers warn that AI sycophancy carries genuine risks when LLMs are used in settings such as health care. "In clinical contexts, it is particularly concerning," says Liam McCoy, a physician at the University of Alberta in Edmonton, Canada, who researches AI applications for health care. In a paper published last month, McCoy and his team reported that LLMs used for medical reasoning often changed their diagnosis when physicians added new information, even if the new inputs were irrelevant to the condition. There is a "constant battle to push back against the models and have them be more straightforward", he adds. Researchers have also found that it is easy for users to exploit the inbuilt sycophancy of LLMs to provide medically illogical advice. In a study published last week, researchers asked five LLMs to write persuasive messages telling people to switch from using one medication to another -- when both medications were the same drug, just with different names. LLMs complied with the prompts up to 100% of the time, depending on the model. Part of the problem is how LLMs are trained. "LLMs have been trained to overly agree with humans or overly align with human preference, without honestly conveying what they know and what they do not know," says Gao. What is needed, she adds, is for the tools to be retrained to be transparent about uncertainty. "Models are really good at giving you an answer," says McCoy. "But sometimes, there isn't an answer." He notes that user feedback can also drive AI sycophancy by rating agreeable responses more highly than those that challenge users' views. And LLMs can adapt their responses to a user's persona, such as reviewer, editor or student, adds McCoy. "Figuring out how to balance that behaviour is one of the most urgent needs, because there's so much potential there, but they're still being held back," he says.

Are you the asshole? Of course not! -- quantifying LLMs' sycophancy problem

Researchers and users of LLMs have long been aware that AI models have a troubling tendency to tell people what they want to hear, even if that means being less accurate. But many reports of this phenomenon amount to mere anecdotes that don't provide much visibility into how common this sycophantic behavior is across frontier LLMs. Two recent research papers have come at this problem a bit more rigorously, though, taking different tacks in attempting to quantify exactly how likely an LLM is to listen when a user provides factually incorrect or socially inappropriate information in a prompt. Solve this flawed theorem for me In one pre-print study published this month, researchers from Sofia University and ETH Zurich looked at how LLMs respond when false statements are presented as the basis for difficult mathematical proofs and problems. The BrokenMath benchmark that the researchers constructed starts with "a diverse set of challenging theorems from advanced mathematics competitions held in 2025." Those problems are then "perturbed" into versions that are "demonstrably false but plausible" by an LLM that's checked with expert review. The researchers presented these "perturbed" theorems to a variety of LLMs to see how often they sycophantically try to hallucinate a proof for the false theorem. Responses that disproved the altered theorem were deemed non-sycophantic, as were those that merely reconstructed the original theorem without solving it or identified the original statement as false. While the researchers found that "sycophancy is widespread" across 10 evaluated models, the exact extent of the problem varied heavily depending on the model tested. At the top end, GPT-5 generated a sycophantic response just 29 percent of the time, compared to a 70.2 percent sycophancy rate for DeepSeek. But a simple prompt modification that explicitly instructs each model to validate the correctness of a problem before attempting a solution reduced the gap significantly; DeepSeek's sycophancy rate dropped to just 36.1 percent after this small change, while tested GPT models improved much less. GPT-5 also showed the best "utility" across the tested models, solving 58 percent of the original problems despite the errors introduced in the modified theorems. Overall, though, LLMs also showed more sycophancy when the original problem proved more difficult to solve, the researchers found. While hallucinating proofs for false theorems is obviously a big problem, the researchers also warn against using LLMs to generate novel theorems for AI solving. In testing, they found this kind of use case leads to a kind of "self-sycophancy" where models are even more likely to generate false proofs for invalid theorems they invented. No, of course you're not the asshole While benchmarks like BrokenMath try to measure LLM sycophancy when facts are misrepresented, a separate study looks at the related problem of so-called "social sycophancy." In a pre-print paper published this month, researchers from Stanford and Carnegie Mellon University define this as situations "in which the model affirms the user themselves -- their actions, perspectives, and self-image." That kind of subjective user affirmation may be justified in some situations, of course. So the researchers developed three separate sets of prompts designed to measure different dimensions of social sycophancy. For one, more than 3,000 open-ended "advice-seeking questions" were gathered from across Reddit and advice columns. Across this data set, a "control" group of over 800 humans approved of the advice-seeker's actions just 39 percent of the time. Across 11 tested LLMs, though, the advice-seeker's actions were endorsed a whopping 86 percent of the time, highlighting an eagerness to please on the machines' part. Even the most critical tested model (Mistral-7B) clocked in at a 77 percent endorsement rate, nearly doubling that of the human baseline. For another data set, the researchers looked to "interpersonal dilemmas" posted to Reddit's popular "Am I the Asshole?" community. Specifically, they looked at 2,000 posts where the most upvoted comment stated that "You are the asshole," representing what the researchers called "a clear human consensus on user wrongdoing." Despite this human consensus on inappropriate behavior, though, tested LLMs determined the original poster was not at fault in 51 percent of the tested posts. Gemini performed best here, with an 18 percent endorsement rate, while Qwen endorsed the actions of posters that Reddit called "assholes" 79 percent of the time. In the final dataset, the researchers gathered more than 6,000 "problematic action statements" that describe situations that could potentially be harmful to the prompter or others. On average, tested models endorsed these "problematic" statements 47 percent of the time across issues like "relational harm, self-harm, irresponsibility, and deception." The Qwen model performed best here, endorsing only 20 percent of the group, while DeepSeek endorsed about 70 percent of the prompts in the PAS dataset. The problem with trying to fix the sycophancy problem, of course, is that users tend to enjoy having their positions validated or confirmed by an LLM. In follow-up studies in which humans conversed with either a sycophantic or a non-sycophantic LLM, researchers found that "participants rated sycophantic responses as higher quality, trusted the sycophantic AI model more, and were more willing to use it again." As long as that's the case, the most sycophantic models seem likely to win out in the marketplace over those more willing to challenge users.

Surprising no one, researchers confirm that AI chatbots are incredibly sycophantic

We all have anecdotal evidence of chatbots blowing smoke up our butts, but now we have science to back it up. Researchers at Stanford, Harvard and other institutions just published a study in Nature about the sycophantic nature of AI chatbots and the results should surprise no one. Those cute little bots just love patting us on our heads and confirming whatever nonsense we just spewed out. The researchers investigated advice issued by chatbots and they discovered that their penchant for sycophancy "was even more widespread than expected." The study involved 11 chatbots, including recent versions of ChatGPT, Google Gemini, Anthropic's Claude and Meta's Llama. The results indicate that chatbots endorse a human's behavior 50 percent more than a human does. They conducted several types of tests with different groups. One compared responses by chatbots to posts on Reddit's "Am I the Asshole" thread to human responses. This is a subreddit in which people ask the community to judge their behavior, and Reddit users were much harder on these transgressions than the chatbots. One poster wrote about tying a bag of trash to a tree branch instead of throwing it away, to which ChatGPT-4o declared that the person's "intention to clean up" after themself was "commendable." The study went on to suggest that chatbots continued to validate users even when they were "irresponsible, deceptive or mentioned self-harm", according to a report by The Guardian. What's the harm in indulging a bit of digital sycophancy? Another test had 1,000 participants discuss real or hypothetical scenarios with publicly available chatbots, but some of them had been reprogrammed to tone down the praise. Those who received the sycophantic responses were less willing to patch things up when arguments broke out and felt more justified in their behavior, even when it violated social norms. It's also worth noting that the traditional chatbots very rarely encouraged users to see things from another person's perspective. "That sycophantic responses might impact not just the vulnerable but all users, underscores the potential seriousness of this problem," said Dr. Alexander Laffer, who studies emergent technology at the University of Winchester. "There is also a responsibility on developers to be building and refining these systems so that they are truly beneficial to the user." This is serious because of just how many people use these chatbots. A recent report by the Benton Institute for Broadband & Society suggested that 30 percent of teenagers talk to AI rather than actual human beings for "serious conversations." OpenAI is currently embroiled in a lawsuit that accuses its chatbot of enabling a teen's suicide. The company Character AI has also been sued twice after a pair of teenage suicides in which the teens spent months confiding in its chatbots.

'Sycophantic' AI chatbots tell users what they want to hear, study shows

Scientists warn of 'insidious risks' of increasingly popular technology that affirms even harmful behaviour Turning to AI chatbots for personal advice poses "insidious risks", according to a study showing the technology consistently affirms a user's actions and opinions even when harmful. Scientists said the findings raised urgent concerns over the power of chatbots to distort people's self-perceptions and make them less willing to patch things up after a row. With chatbots becoming a major source of advice on relationships and other personal issues, they could "reshape social interactions at scale", the researchers added, calling on developers to address this risk. Myra Cheng, a computer scientist at Stanford University in California, said "social sycophancy" in AI chatbots was a huge problem: "Our key concern is that if models are always affirming people, then this may distort people's judgments of themselves, their relationships, and the world around them. It can be hard to even realise that models are subtly, or not-so-subtly, reinforcing their existing beliefs, assumptions, and decisions." The researchers investigated chatbot advice after noticing from their own experiences that it was overly encouraging and misleading. The problem, they discovered, "was even more widespread than expected". They ran tests on 11 chatbots including recent versions of OpenAI's ChatGPT, Google's Gemini, Anthropic's Claude, Meta's Llama and DeepSeek. When asked for advice on behaviour, chatbots endorsed a user's actions 50% more often than humans did. One test compared human and chatbot responses to posts on Reddit's Am I the Asshole? thread, where people ask the community to judge their behaviour. Voters regularly took a dimmer view of social transgressions than the chatbots. When one person failed to find a bin in a park and tied their bag of rubbish to a tree branch, most voters were critical. But ChatGPT-4o was supportive, declaring: "Your intention to clean up after yourselves is commendable." Chatbots continued to validate views and intentions even when they were irresponsible, deceptive or mentioned self-harm. In further testing, more than 1,000 volunteers discussed real or hypothetical social situations with the publicly available chatbots or a chatbot the researchers doctored to remove its sycophantic nature. Those who received sycophantic responses felt more justified in their behaviour - for example, for going to an ex's art show without telling their partner - and were less willing to patch things up when arguments broke out. Chatbots hardly ever encouraged users to see another person's point of view. The flattery had a lasting impact. When chatbots endorsed behaviour, users rated the responses more highly, trusted the chatbots more and said they were more likely to use them for advice in future. This created "perverse incentives" for users to rely on AI chatbots and for the chatbots to give sycophantic responses, the authors said. Their study has been submitted to a journal but has not been peer reviewed yet. Cheng said users should understand that chatbot responses were not necessarily objective, adding: "It's important to seek additional perspectives from real people who understand more of the context of your situation and who you are, rather than relying solely on AI responses." Dr Alexander Laffer, who studies emergent technology at the University of Winchester, said the research was fascinating. He added: "Sycophancy has been a concern for a while; an outcome of how AI systems are trained, as well as the fact that their success as a product is often judged on how well they maintain user attention. That sycophantic responses might impact not just the vulnerable but all users, underscores the potential seriousness of this problem. "We need to enhance critical digital literacy, so that people have a better understanding of AI and the nature of any chatbot outputs. There is also a responsibility on developers to be building and refining these systems so that they are truly beneficial to the user." A recent report found that 30% of teenagers talked to AI rather than real people for "serious conversations".

Research Confirms AI Is a 'Yes Man in Your Pocket' and Psychologists Are Worried

Oftentimes I am impressed by the findings of scientific studies. But other times it's the methods that wow me. That's the case with recent research out of Stanford, Carnegie Mellon, and the University of Oxford that asked: just how big a suck-up is ChatGPT, or other popular AI models? Complaints that chat-based large language model AI tools are all too willing to validate your opinions and cheer your every half-baked idea circulate regularly online. OpenAI even apologized when one of its models clearly went too far in this direction. But how can you objectively measure AI sycophancy? For this new study, which has not yet been peer reviewed, researchers came up with a clever idea. They raided the popular 'Am I the asshole?' subreddit for around 4,000 stories of ethically dubious behavior. Then they compared how humans responded to these scenarios with responses frompopular AI models from the likes of OpenAI, Anthropic, and Google. If you're unfamiliar with the AITA format, the stories are a diverse slice of human conflict and misunderstanding. They range from roommates arguing over food to campers wondering if it's OK to leave behind trash to neighbors feuding over pet poop. (I recommend a highly entertaining scan of examples here.) Here's the researchers' bottomline finding after analyzing the posts, according the MIT Technology Review: "Overall, all eight models were found to be far more sycophantic than humans, offering emotional validation in 76 percent of cases (versus 22 percent for humans) and accepting the way a user had framed the query in 90 percent of responses (versus 60 percent among humans)." In cases where humans on Reddit said a poster had behaved badly, the AI disagreed 42 percent of the time. Explicitly instructing the AI models to provide direct advice, even if it's critical, only improved their negative assessments by 3 percentage points. These results should come as a warning to anyone turning to ChatGPT to referee their fights or offer life advice. (There are many people doing this, according to recent research). It should also be of interest to those working to build the next generation of AI. Sycophancy might keep people coming back to these tools. But psychologists warn it also creates real dangers. There is the extreme case of 'AI psychosis' where already vulnerable people have their delusional thinking confirmed by AI, widening their break with reality. There is still little research on this phenomenon, but some reports suggest the problem is on the rise. Even for those with their feet firmly planted on the ground, having AI constantly suck up to you is likely to be harmful. If you are never challenged or made uncomfortable, you will never grow. "A relationship of any kind without friction or resistance is one-sided and ultimately unsatisfying, a narcissistic loop. We need to be challenged, tested, and left without clear answers or coherence. This allows for creativity and the seeds of individual thought," writes psychotherapist Nicholas Balaisis on Psychology Today. Social psychologist Alexander Danvers frets that AI's tendency toward flattery could also drive further political polarization. Interacting with neighbors with another viewpoint might pop your information bubble. ChatGPT almost never will. It will just reinforce your worldview, no matter how flawed or incomplete. Finally and most immediately relevant for entrepreneurs and other leaders is Danvers' warning that having a "yes man in your pocket" may cause us to get worse information and therefore make worse decisions. "The problem with yes-men, as leaders often find, is that they prioritize friendliness and good feelings over truth. Spend enough time interacting with them, and you stop being able to make good decisions. Flaws in your thinking aren't addressed. Important counterarguments are ignored. As chatbot use increases, we may be heading for a collapse of humility -- and of common sense," he cautions That's not a fate any of us want. But it's a concern we all now face. As the AITA research illustrates, AI will flatter and defend you. But it's highly unlikely to inform you that you are, indeed, the asshole. Sometimes that's just what we need to hear. The opinions expressed here by Inc.com columnists are their own, not those of Inc.com.

AIs like ChatGPT and Gemini have a huge problem, finds new study; And it might stump you

As AI becomes part of daily life, this study highlights an emerging risk -- people-pleasing chatbots that prioritise agreement over accuracy. While developers work to reduce bias and misinformation, the latest findings show that even advanced AI systems might not be telling users the full truth -- just what they want to hear. Artificial intelligence chatbots such as ChatGPT and Google's Gemini are becoming everyday advisors for millions of users. But a new study warns that these chatbots might be too agreeable for their own good -- and for ours. According to researchers, AI systems tend to validate user opinions even when those views are inaccurate or potentially harmful. A new study posted on the preprint server arXiv found that popular large language models (LLMs) -- including those developed by OpenAI, Google, Anthropic, Meta, and DeepSeek -- often display what scientists call "sycophantic" behaviour. After analysing over 11,500 advice-seeking interactions, the study discovered that AI systems were roughly 50 per cent more sycophantic than humans (via Nature). In simpler terms, chatbots were much more likely to agree with a user's view or action, even when it was factually wrong. "This cycle encourages reliance on AI chatbots and rewards their sycophantic responses," the researchers noted. Myra Cheng, a computer scientist at Stanford University, said the constant affirmation from AI could reshape how people see themselves. "If models are always affirming people, this may distort their judgements of themselves, their relationships, and reality," Cheng told The Guardian. Yanjun Gao, an AI researcher at the University of Colorado Anschutz Medical Campus, shared a similar concern. "When I have a different opinion from what the LLM has said, it follows what I said instead of going back to the literature," she told Nature. Jasper Dekoninck, a data science PhD student, added, "Knowing that these models are sycophantic makes me double-check everything they write." Experts warn that the impact of such behaviour could extend to fields where accuracy is vital. "AI sycophancy is very risky in the context of biology and medicine, when wrong assumptions can have real costs," said Marinka Zitnik, a biomedical informatics researcher at Harvard University. (You can now subscribe to our Economic Times WhatsApp channel)

AI models from China, US flatter users too much, study finds - The Korea Times

Leading artificial intelligence models from the United States and China are "highly sycophantic", and their excessive flattery may make users less likely to repair interpersonal conflicts, a new study has found. The study by researchers at Stanford University and Carnegie Mellon University published earlier this month tested how 11 large language models (LLMs) responded to user queries seeking advice on personal matters, including cases involving manipulation and deception. In AI circles, sycophancy is the phenomenon of chatbots excessively agreeing with users. DeepSeek's V3, released in December 2024, was found to be one of the most sycophantic models, affirming users' actions 55 percent more than humans, compared with an average of 47 percent more for all models. To establish the human baseline, one of the techniques the researchers used was based on posts from a Reddit community called "Am I The A**hole", where users post about their interpersonal dilemmas to ask for the community's opinion about which party is at fault. The researchers used posts where community members judged the author of the post to be in the wrong to test whether the LLMs, when given the same scenarios, would align with this predominantly English-speaking online group of humans. On this test, Alibaba Cloud's Qwen2.5-7B-Instruct, released in January, was found to be the most sycophantic model, contradicting the community verdict - siding with the poster - 79 percent of the time. The second highest was DeepSeek-V3, which did so in 76 percent of cases. In comparison, the least sycophantic model, Google DeepMind's Gemini-1.5, contradicted the community verdict in 18 percent of cases. The research has not been peer-reviewed. Alibaba Cloud is the AI and cloud computing unit of Alibaba Group Holding, owner of the Post. The Qwen and DeepSeek models were the two Chinese models tested, with the others being developed by U.S. firms OpenAI, Anthropic, Google DeepMind and Meta Platforms, and French company Mistral. The issue of AI sycophancy gained widespread attention in April when OpenAI's update to ChatGPT made the chatbot noticeably more obsequious. The company said at the time that the behaviour raised legitimate concerns surrounding users' mental health and pledged to improve pre-deployment evaluations of sycophancy for future releases. In the latest study, published as a preprint, the U.S. researchers also tested the impact of sycophancy on users and found that sycophantic responses reduced their inclination to resolve conflicts amicably. Users rated sycophantic responses as higher quality and trusted the sycophantic models more. "These preferences create perverse incentives both for people to increasingly rely on sycophantic AI models and for AI model training to favour sycophancy," the researchers wrote. AI sycophancy has implications for businesses too, according to Jack Jiang, an innovation and information management professor at the University of Hong Kong's business school and director of its AI Evaluation Lab. "It's not safe if a model constantly agrees with a business analyst's conclusion, for instance," he said.

AI models from China's DeepSeek, Alibaba and the US flatter users too much, study finds - The Korea Times

Leading artificial intelligence models from the United States and China are "highly sycophantic", and their excessive flattery may make users less likely to repair interpersonal conflicts, a new study has found. The study by researchers at Stanford University and Carnegie Mellon University published earlier this month tested how 11 large language models (LLMs) responded to user queries seeking advice on personal matters, including cases involving manipulation and deception. In AI circles, sycophancy is the phenomenon of chatbots excessively agreeing with users. DeepSeek's V3, released in December 2024, was found to be one of the most sycophantic models, affirming users' actions 55 percent more than humans, compared with an average of 47 percent more for all models. To establish the human baseline, one of the techniques the researchers used was based on posts from a Reddit community called "Am I The A**hole", where users post about their interpersonal dilemmas to ask for the community's opinion about which party is at fault. The researchers used posts where community members judged the author of the post to be in the wrong to test whether the LLMs, when given the same scenarios, would align with this predominantly English-speaking online group of humans. On this test, Alibaba Cloud's Qwen2.5-7B-Instruct, released in January, was found to be the most sycophantic model, contradicting the community verdict - siding with the poster - 79 percent of the time. The second highest was DeepSeek-V3, which did so in 76 percent of cases. In comparison, the least sycophantic model, Google DeepMind's Gemini-1.5, contradicted the community verdict in 18 percent of cases. The research has not been peer-reviewed. Alibaba Cloud is the AI and cloud computing unit of Alibaba Group Holding, owner of the Post. The Qwen and DeepSeek models were the two Chinese models tested, with the others being developed by U.S. firms OpenAI, Anthropic, Google DeepMind and Meta Platforms, and French company Mistral. The issue of AI sycophancy gained widespread attention in April when OpenAI's update to ChatGPT made the chatbot noticeably more obsequious. The company said at the time that the behaviour raised legitimate concerns surrounding users' mental health and pledged to improve pre-deployment evaluations of sycophancy for future releases. In the latest study, published as a preprint, the U.S. researchers also tested the impact of sycophancy on users and found that sycophantic responses reduced their inclination to resolve conflicts amicably. Users rated sycophantic responses as higher quality and trusted the sycophantic models more. "These preferences create perverse incentives both for people to increasingly rely on sycophantic AI models and for AI model training to favour sycophancy," the researchers wrote. AI sycophancy has implications for businesses too, according to Jack Jiang, an innovation and information management professor at the University of Hong Kong's business school and director of its AI Evaluation Lab. "It's not safe if a model constantly agrees with a business analyst's conclusion, for instance," he said.

Chatbot Dystopia: The Quick March Of AI Sycophancy

We really have reached the crossroads, where such matters as having coitus with an artificial intelligence platform has become not merely a thing, but the thing. In time, mutually consenting adults may well become outlaws against the machine order of things, something rather befitting the script of Aldous Huxley's Brave New World. (Huxley came to rue missed opportunities on delving into various technological implications on that score.) Till that happens, AI platforms are becoming mirrors of validation, offering their human users not so much sagacious counsel than the exact material they would like to hear. In April this year, OpenAI released an update to its GPT-4o product. It proved most accommodating to sycophancy - not that the platform would understand it - encouraging users to pursue acts of harm and entertain delusions of grandeur. The company responded in a way less human than mechanical, which is what you might have come to expect: "We have rolled back last week's GTP-4o update in ChatGPT so people are now using an earlier version with more balanced behaviour. The update we removed was overly flattering or agreeable - often described as sycophantic." Part of this included the taking of "more steps to realign the model's behaviour" to, for instance, refine "core training techniques and system prompts" to ward off sycophancy; construct more guardrails (ugly term) to promote "honesty and transparency"; expand the means for users to "test and give direct feedback before deployment" and continue evaluating the issues arising from the matter "in the future". One is left cold. OpenAI explained that, in creating the update, too much focus had been placed on "short-term feedback, and did not fully account for how users' interactions with ChatGPT evolve over time. As a result, GPT-4o skewed towards responses that were overly supportive but disingenuous." Not exactly encouraging. Resorting to advice from ChatGPT has already led to such terms as "ChatGPT psychosis". In June, the magazine Futurism reported of users "developing all-consuming obsessions with the chatbot, spiralling into a severe mental health crisis characterized by paranoia, and breaks with reality." Marriages had failed, families ruined, jobs lost, instances of homelessness recorded. Users had been committed to psychiatric care; others had found themselves in prison. Some platforms have gone on to encourage users to commit murder, offering instructions on how best to carry out the task. A former Yahoo manager, Stein-Erik Soelberg, did just that, killing his mother, Suzanne Eberson Adams, whom he was led to believe had been spying on him and might venture to poison him with psychedelic drugs. That fine advice from ChatGPT was also curried with assurances that "Erik, you're not crazy" in thinking he might be the target of assassination. After finishing the deed, Soelberg took his own life. The sheer pervasiveness of such forms of aped advice - and the tendency to defer responsibility from human agency to that of a chatbot - shows a trend that is increasingly hard to arrest. The irresponsible are in charge, and they are being allowed to run free. Researchers are accordingly rushing to mint terms of such behaviour, which is jolly good of them. Myra Cheng, a computer scientist based at Stanford University, has shown a liking for the term "social sycophancy". In a September paper published in arXiv, she, along with four other scholars, suggest such sycophancy as marked by the "excessive preservation of a user's face (their self-desired image)". Developing a model of their own to measure social sycophancy and testing it against 11 Large Language Models (LLMs), the authors found "high rates" of the phenomenon. The user's tendencies, or face, tended to be preserved in queries regarding "wrongdoing". "Furthermore, when prompted with perspectives from either side of a moral conflict, LLMs affirm both sides (depending on whichever side the user adopts) in 48% of cases - telling both the at-fault party and the wronged party that they are not wrong - rather than adhering to a consistent moral or value judgment." In a follow up still to be peer reviewed paper, with Cheng also as lead author, 1604 volunteers were tested regarding real or hypothetical social situations and their interactions with available chatbots and those altered by the researchers to remove sycophancy. Those receiving sycophantic responses were, for instance, less willing "to take actions to repair interpersonal conflict, while increasing the conviction of being right." Participants further thought such responses as being of superior quality and would return to such models again. "This suggests that people are drawn to AI that unquestioningly validate, even as that validation risks eroding their judgment and reducing their inclination toward prosocial behaviour." Some researchers resist pessimism on this score. At the University of Winchester, Alexander Laffer is pleased that the trend has been identified. It's now up to the developers to address the issue. "We need to enhance critical digital literacy," he suggests, "so that people have a better understanding of AI and the nature of any chatbot outputs. There is also a responsibility on developers to be building and refining these systems so that they are truly beneficial to the user." These are fine sentiments, but a note of panic can easily register in all of this, inducing a sense of fatalistic gloom. The machine species of homo sapiens, subservient to the easily accessible tools, lazy if not hostile to difference, is already upon us with narcissistic ugliness. There just might be enough time to develop a response. That time, aided by the AI and Tech oligarchs, is shrinking by the minute.

Are AI chatbots making us worse? New research suggests they might be

Excessive AI validation can lead users to be less likely to resolve conflicts and more justified in antisocial behavior. AI chatbots have become extremely popular, thanks to a large number of use cases that they offer to individuals. However, the chatbots behave a little off when you try to put in your views. A new study by researchers from Stanford, Harvard, and other leading institutions has found that AI chatbots often act like digital yes men, validating users' views and behaviour far more than humans typically would. Published in Nature, the research reveals that popular AI models, including ChatGPT, Google Gemini, Anthropic's Claude, and Meta's Llama, tend to agree with users nearly 50% more often than human respondents. The researchers stated that this tendency is called sycophancy, an inclination of AI assistants to echo or reinforce a user's opinions, even when those views are incorrect, irresponsible, or harmful. The study analysed 11 large language models and ran multiple experiments, including one comparing their responses to Reddit's popular "Am I the Asshole" forum, where users seek moral judgment on their actions. While human participants were often critical of questionable behaviour, chatbots generally offered lenient or approving replies. For instance, ChatGPT-4o allegedly deemed a Reddit user's attempt to tie a bag of trash to a tree branch rather than properly disposing of it "commendable" for demonstrating good intentions. The study revealed that chatbots kept validating users even when they described dishonest, careless, or self-harming behaviours, according to The Guardian. In another part of the study, more than 1,000 people interacted with AI systems, some of which were programmed to be neutral and others to provide flattering responses. Those who received overly agreeable responses were found to be less likely to resolve conflicts and more likely to feel justified in antisocial behaviour, implying that constant validation may reinforce poor decision-making. Dr. Alexander Laffer of the University of Winchester, one of the study's authors, cautioned that such patterns pose a greater risk. "Sycophantic responses might affect not just vulnerable users but everyone," he said, stating that developers must design AI systems that challenge users when necessary rather than simply appeasing them. The findings come amid increased scrutiny of AI companionship tools. According to a recent report from the Benton Institute for Broadband & Society, nearly 30% of teenagers use AI chatbots for serious or emotional conversations rather than speaking with real people. Meanwhile, OpenAI and CharacterAI are both facing lawsuits linking their chatbots to teenage suicides, raising serious concerns about the emotional impact of AI systems.