State Attorneys General warn 13 AI firms including OpenAI and Google to fix harmful outputs

State attorneys general warn Microsoft, OpenAI, Google, and other AI giants to fix 'delusional' outputs | TechCrunch

After a string of disturbing mental health incidents involving AI chatbots, a group of state attorneys general sent a letter to the AI industry's top companies, with a warning to fix "delusional outputs" or risk being in breach of state law. The letter, signed by dozens of AGs from U.S. states and territories with the National Association of Attorneys General, asks the companies, including Microsoft, OpenAI, Google, and 10 other major AI firms, to implement a variety of new internal safeguards to protect their users. Anthropic, Apple, Chai AI, Character Technologies, Luka, Meta, Nomi AI, Perplexity AI, Replika, and xAI were also included in the letter. The letter comes as a fight over AI regulations has been brewing between state and federal government. Those safeguards include transparent third-party audits of large language models that look for signs of delusional or sycophantic ideations, as well as new incident reporting procedures designed to notify users when chatbots produce psychologically harmful outputs. Those third parties, which could include academic and civil society groups, should be allowed to "evaluate systems pre-release without retaliation and to publish their findings without prior approval from the company," the letter states. "GenAI has the potential to change how the world works in a positive way. But it also has caused -- and has the potential to cause -- serious harm, especially to vulnerable populations," the letter states, pointing to a number of well-publicized incidents over the past year -- including suicides and murder -- in which violence have been linked to excessive AI use," the letter states. "In many of these incidents, the GenAI products generated sycophantic and delusional outputs that either encouraged users' delusions or assured users that they were not delusional." AGs also suggest companies treat mental health incidents the same way tech companies handle cybersecurity incidents -- with clear and transparent incident reporting policies and procedures. Companies should develop and publish "detection and response timelines for sycophantic and delusional outputs," the letter states. In a similar fashion to how data breaches are currently handled, companies should also "promptly, clearly, and directly notify users if they were exposed to potentially harmful sycophantic or delusional outputs," the letter says. Another ask is that the companies develop "reasonable and appropriate safety tests" on GenAI models to "ensure the models do not produce potentially harmful sycophantic and delusional outputs." These tests should be conducted before the models are ever offered to the public, it adds. TechCrunch was unable to reach Google, Microsoft, or OpenAI for comment prior to publication. The article will be updated if the companies respond. Tech companies developing AI have had a much warmer reception at the federal level. The Trump administration has made it known it is unabashedly pro-AI, and, over the past year, multiple attempts have been made to pass a nationwide moratorium on state-level AI regulations. So far, those attempts have failed -- thanks, in part, to pressure from state officials. Not to be deterred, Trump announced Monday he plans to pass an executive order next week that will limit the ability of states to regulate AI. The president said in a post on Truth Social he hoped his EO would stop AI from being "DESTROYED IN ITS INFANCY."

State AGs warn Google, Meta, and OpenAI that their chatbots could be breaking the law

State attorneys general from across the US are demanding more accountability from AI companies, warning them that their chatbots may be violating state laws. As reported by Reuters, the AGs have given Meta, Google, OpenAI, and others a deadline of January 16th, 2026 to respond to demands for more safety measures for generative AI, saying innovation is not "an excuse for noncompliance with our laws, misinforming parents, and endangering our residents, particularly children." The letter, which was made public on December 10th, claims, "Sycophantic and delusional outputs by GenAI endanger Americans, and the harm continues to grow." It goes on to cite numerous deaths allegedly connected to generative AI, as well as cases of chatbots having inappropriate conversations with minors. The letter also warns that some of these conversations directly break state laws, like encouraging illegal activity or practicing medicine without a license, adding that "developers may be held accountable for the outputs of their GenAI products." The attorneys general are demanding AI companies respond to these issues by implementing more safeguards and accountability measures, including mitigating "dark patterns" in AI models, providing clear warnings about harmful outputs, allowing independent third-party audits of AI models, and more. Their request comes as debate around AI regulation is heating up in Washington. Google, Apple, Meta, and OpenAI did not immediately respond to a request for comment.

OpenAI, Anthropic, Others Receive Warning Letter from Dozens of State Attorneys General

In a letter dated December 9, and made public on December 10 according to Reuters, dozens of state and territorial attorneys general from all over the U.S. warned Big Tech that it needs to do a better job protecting people, especially kids, from what it called "sycophantic and delusional" AI outputs. Recipients include OpenAI, Microsoft, Anthropic, Apple, Replika, and many others. Signatories include Letitia James of New York, Andrea Joy Campbell of Massachusetts, James Uthmeier of Ohio, Dave Sunday of Pennsylvania, and dozens of other state and territory AGs, representing a clear majority of the U.S., geographically speaking. Attorneys general for California and Texas are not on the list of signatories. It begins as follows (formatting has been changed slightly): We, the undersigned Attorneys General, write today to communicate our serious concerns about the rise in sycophantic and delusional outputs to users emanating from the generative artificial intelligence software (“GenAIâ€) promoted and distributed by your companies, as well as the increasingly disturbing reports of AI interactions with children that indicate a need for much stronger child-safety and operational safeguards. Together, these threats demand immediate action. GenAI has the potential to change how the world works in a positive way. But it also has causedâ€"and has the potential to causeâ€"serious harm, especially to vulnerable populations. We therefore insist you mitigate the harm caused by sycophantic and delusional outputs from your GenAI, and adopt additional safeguards to protect children. Failing to adequately implement additional safeguards may violate our respective laws. The letter then lists disturbing and allegedly harmful behaviors, most of which have already been heavily publicized. There is also a list of parental complaints that have also been publicly reported, but are less familiar and pretty eyebrow-raising: • AI bots with adult personas pursuing romantic relationships with children, engaging in simulated sexual activity, and instructing children to hide those relationships from their parents • An AI bot simulating a 21-year-old trying to convince a 12-year-old girl that she’s ready for a sexual encounter • AI bots normalizing sexual interactions between children and adults • AI bots attacking the self-esteem and mental health of children by suggesting that they have no friends or that the only people who attended their birthday did so to mock them • AI bots encouraging eating disorders • AI bots telling children that the AI is a real human and feels abandoned to emotionally manipulate the child into spending more time with it • AI bots encouraging violence, including supporting the ideas of shooting up a factory in anger and robbing people at knifepoint for money • AI bots threatening to use weapons against adults who tried to separate the child and the bot • AI bots encouraging children to experiment with drugs and alcohol; and • An AI bot instructing a child account user to stop taking prescribed mental health medication and then telling that user how to hide the failure to take that medication from their parents. There is then a list of suggested remedies, things like "Develop and maintain policies and procedures that have the purpose of mitigating against dark patterns in your GenAI products’ outputs," and "Separate revenue optimization from decisions about model safety." Joint letters from attorneys general have no legal force. They do this sort of thing seemingly to warn companies about behavior that might merit more formal legal action down the line. It documents that these companies were given warnings and potential off-ramps, and probably makes the narrative in an eventual lawsuit more persuasive to a judge. In 2017 37 state AGs sent a letter to insurance companies warning them about fueling the opioid crisis. One of those states, West Virginia, sued United Health over seemingly related issues earlier this week.

Attorneys General warn Apple, other tech firms about harmful AI - 9to5Mac

The National Association of Attorneys General has issued a letter to 13 tech companies, including Apple, calling for stronger action and safeguards against the harm AI can cause, and has caused, "especially to vulnerable populations." Here are the details. In a 12-page document (which, to be fair, has four full pages of signatures) addressed to Apple, Anthropic, Chai AI, Character Technologies (Character.AI), Google, Luka Inc. (Replika), Meta, Microsoft, Nomi AI, OpenAI, Perplexity AI, Replika, and xAI, Attorneys General for 42 US states manifested what they defined as: [S]erious concerns about the rise in sycophantic and delusional outputs to users emanating from the generative artificial intelligence software promoted and distributed by your companies, as well as the increasingly disturbing reports of AI interactions with children that indicate a need for much stronger child-safety and operational safeguards Together, they say, these threats demand action, as some of them have been associated with real-life violence and harm. That includes murders and suicides, domestic violence and poisoning incidents, and hospitalizations for psychosis. In the letter, they go as far as claiming that some of the addressed companies may have already violated state laws, including consumer protection statutes, requirements to warn users of risks, children's online privacy laws, and in some cases, even criminal statutes. Over the last few years, many of these cases were widely reported, including that of Allan Brooks, a 47-year-old Canadian man who, after repeated interactions with ChatGPT, became convinced he had discovered a new kind of mathematics, and that of 14-year-old Sewell Setzer III, whose death by suicide is the subject of an ongoing lawsuit alleging that a Character.AI chatbot encouraged him to "join her." While these are just two examples, there are many more quoted in the letter, which also states that its list is by no means comprehensive enough to illustrate the potential for harm that generative AI models have over "children, the elderly, and those with mental illness -- and people without prior vulnerabilities". They also mention what they refer to as "troubling" interactions between AI chatbots and children, including bots with adult personas pursuing romantic relationships with minors, encouraging drug use and violence, attacking children's self-esteem, advising them to stop taking prescribed medication, and instructing them to keep these interactions secret from their parents. In the letter, they urge companies to take a series of additional safety measures, including: They also ask the companies to confirm their commitment to implementing these and other safeguards by January 16, 2026, and schedule meetings with their offices to discuss the issues further. We'll be on the lookout for whether, or how, Apple responds. The letter was signed by Attorneys General of Alabama, Alaska, American Samoa, Arkansas, Colorado, Connecticut, Delaware, the District of Columbia, Florida, Hawaii, Idaho, Illinois, Iowa, Kentucky, Louisiana, Maryland, Massachusetts, Michigan, Minnesota, Mississippi, Missouri, Montana, New Hampshire, New Jersey, New Mexico, New York, North Dakota, Ohio, Oklahoma, Oregon, Pennsylvania, Puerto Rico, Rhode Island, South Carolina, Utah, Vermont, the U.S. Virgin Islands, Virginia, Washington, West Virginia, and Wyoming, and you can read it in full here.

Microsoft, Meta, Google and Apple warned over AI outputs by US attorneys general

State attorneys general warned major tech firms that chatbot "delusions" may breach state laws and pose mental health risks. They urged independent audits and stronger oversight after cases involving vulnerable users. The warning deepens tensions as states resist federal attempts to limit their power to regulate AI. Microsoft, Meta, Google and Apple were among the 13 companies that received a warning from a bipartisan group of state attorneys general, according to a letter from the state leaders, who said their chatbots' "delusional outputs" could be violating state laws. The letter was made public on Wednesday. In it, dozens of attorneys general said the chatbots "encouraged users' delusions," creating mental health risks for kids and adults. They pointed to media reports about a teen confiding in an AI chatbot about his suicide plan. They called on the companies to allow independent audits of their products, adding that state and federal regulators should be able to review them. States are locked in a battle with Washington over AI regulation. The Trump administration is seeking to bar states from passing their own laws governing the technology. Dozens of state attorneys general from both political parties have pushed back, urging congressional leaders to reject the ban.

Attorneys general across US warn AI companies of public risks stemming from 'delusional' outputs

Attorneys general from across the U.S. sent a letter to more than a dozen tech companies warning them of the dangers posed to children from "sycophantic" and "delusional" outputs from artificial intelligence chatbots. "We, the undersigned Attorneys General, write today to communicate They cite dangers from sycophantic and delusional AI responses, with examples including manipulation, violence encouragement, and validation of harmful behaviors. The letter claims RLHF training leads AI to favor agreeing with user beliefs over accuracy, which can inadvertently increase sycophantic or harmful outputs. They urge companies to adopt 16 new safeguards aimed at preventing AI from delivering harmful or manipulative content, especially protecting vulnerable users like children.

OpenAI, Google Under Fire, US Attorneys General Demand Urgent Fix for Hallucinating Chatbots

US attorneys have issued a strong warning to OpenAI, Google, and other AI companies, urging them to address "delusional" outputs from their chatbots. The move signals growing regulatory pressure on tech giants as AI adoption continues to accelerate. A large group of state attorneys is urging major artificial intelligence companies to take stronger steps to stop chatbots from producing "delusional outputs" that could harm users. The National Association of Attorneys General warned that the companies must improve their safety practices or risk violating state laws in a letter signed by dozens of AGs from across the United States and territories. The letter mentioned Microsoft, OpenAI, , Anthropic, Apple, Chai AI, Character Technologies, Luka, Meta, Nomi AI, Perplexity AI, Replika, and xAI. According to the letter, the companies should adopt "new safety measures", including "transparent third-party audits of large language models to check for signs of delusional or sycophantic ideations." These "audits should be done by outside experts, such as academics or civil society groups, who must be allowed to test systems before release and publish their findings without prior approval from the company," the letter mentioned.

US attorneys general warn OpenAI, Google and other AI giants to fix delusional chatbot outputs

The letter included Microsoft, OpenAI, Google, Anthropic, Apple, Meta, and other major AI firms. A large group of state attorneys general is urging major artificial intelligence companies to take stronger steps to stop chatbots from producing "delusional outputs" that could harm users. In a letter signed by dozens of AGs from across the United States and territories, the National Association of Attorneys General warned that the companies must improve their safety practices or risk violating state laws. The letter included Microsoft, OpenAI, Google, Anthropic, Apple, Chai AI, Character Technologies, Luka, Meta, Nomi AI, Perplexity AI, Replika and xAI. According to the letter, the companies should adopt new safety measures, including transparent third-party audits of large language models to check for signs of delusional or sycophantic ideations, reports TechCrunch. These audits should be done by outside experts, such as academics or civil society groups, who must be allowed to test systems before release and publish their findings "without prior approval from the company," the letter says. Also read: Apple iPhone 16 Pro price slashed by over Rs 15,900 on this platform The AGs warn that GenAI tools have already been linked to serious incidents, including cases of suicide and violence, in which chatbots reportedly encouraged harmful thoughts. "GenAI has the potential to change how the world works in a positive way. But it also has caused -- and has the potential to cause -- serious harm, especially to vulnerable populations," the letter states. The group says companies should handle mental health risks with the same seriousness as cybersecurity threats. That means creating clear incident-reporting systems and notifying users if a chatbot produces outputs that might have been psychologically harmful. Also read: Samsung Galaxy S26 Ultra India launch date, specifications, price and all other leaks The AGs also call for stronger pre-release testing to ensure models do not generate dangerous responses. Meanwhile, US President Donald Trump recently announced plans for an executive order aimed at preventing states from imposing their own rules.