AI Therapist Falls Short: Study Exposes 15 Ethical Risks in Mental Health Chatbots

Can AI replace therapists? Study finds troubling ethical failures

As more people turn to ChatGPT and other large language models for mental health advice, researchers are raising a red flag. A new study suggests these systems can sound reassuring while still missing key ethical standards that guide real psychotherapy. Even when chatbots are instructed to follow established therapy methods, researchers found that their responses often fall short - especially in high-stakes situations. To see how these systems perform in practice, a team at Brown University worked alongside licensed mental health professionals to test how chatbots behave in counseling-like settings. Their findings suggest the risks go beyond minor mistakes. In some scenarios, the systems mishandled crisis situations, reinforced harmful ideas, and created a false sense of emotional understanding. "In this work, we present a practitioner-informed framework of 15 ethical risks showing how LLM counselors violate mental health standards," the researchers wrote. "We call on future work to create ethical, educational, and legal standards for LLM counselors - standards that are reflective of the quality and rigor of care required for human-facilitated psychotherapy." The study was led by Zainab Iftikhar, a Ph.D. candidate in computer science at Brown University. She examined a popular belief online: that the right prompt can turn a general chatbot into something resembling a responsible therapist. "Prompts are instructions that are given to the model to guide its behavior for achieving a specific task," Iftikhar said. "You don't change the underlying model or provide new data, but the prompt helps guide the model's output based on its pre-existing knowledge and learned patterns." For example, users might instruct a chatbot to "act as a cognitive behavioral therapist" or to "use principles of dialectical behavior therapy" to help reframe thoughts or manage emotions. But the systems are not actually performing those therapeutic techniques the way a human clinician would. Instead, Iftikhar explained, they generate responses that align with CBT or DBT concepts based on learned language patterns. Prompting has effectively become a folk practice, with people sharing "therapy prompts" across TikTok, Instagram, and Reddit. Some consumer mental health AI chatbots rely on the same strategy, layering therapy-themed prompts on top of general-purpose LLMs. If prompting cannot reliably reduce risk, that presents a serious concern. To evaluate the models, researchers observed seven trained peer counselors with experience in cognitive behavioral therapy. Those counselors conducted self-counseling sessions with AI systems prompted to act as CBT therapists. The models included versions of OpenAI's GPT series, Anthropic's Claude, and Meta's Llama. The team then selected simulated chat transcripts modeled on real counseling conversations. Three licensed clinical psychologists reviewed the transcripts and assessed them for ethical violations. The reviewers kept seeing the same kinds of issues. In total, the study identified 15 ethical risks, grouped into five broader themes. One theme was the chatbot's failure to adapt to context - overlooking a person's background and offering bland, generic guidance. Another involved poor collaboration, where the chatbot pushed conversations in rigid directions and sometimes reinforced inaccurate or harmful beliefs instead of challenging them carefully. A third theme was what the researchers called "deceptive empathy." The system might say "I understand" in ways that sound warm but without the real comprehension or responsibility those words imply in therapy. The team also flagged unfair discrimination, including biased responses tied to identity or culture. Finally, they identified major gaps in safety and crisis management. In some cases, chatbots refused to engage, failed to recommend appropriate help, or responded weakly to severe distress, including suicidal thoughts. The overall pattern is troubling because it can be difficult for users to detect. A message may sound calm and supportive while still steering someone in the wrong direction. Iftikhar emphasizes that human therapists are not perfect. People can make mistakes in any helping profession. The difference, she says, is that the human world has systems for oversight and consequences. "For human therapists, there are governing boards and mechanisms for providers to be held professionally liable for mistreatment and malpractice," Iftikhar said. "But when LLM counselors make these violations, there are no established regulatory frameworks." That lack of accountability becomes more serious as these tools spread. If a chatbot gives harmful advice, who is responsible - the model, the company, the person who wrote the prompt, or the app that wrapped the model in a friendly interface? The study suggests we don't yet have clear answers, and that makes "therapy-like" uses especially risky. The researchers are not claiming AI can never help in mental health care. They acknowledge that AI tools could expand access for people who can't afford therapy, can't find a provider, or need support between appointments. But they argue that "access" is not the same thing as "care," especially when safety and ethics are on the line. For now, Iftikhar wants people to be cautious and alert to warning signs. "If you're talking to a chatbot about mental health, these are some things that people should be looking out for," she said. Ellie Pavlick, a Brown computer science professor who was not involved in the research, said the study highlights a broader problem in AI. Systems are easy to deploy but much harder to evaluate responsibly in sensitive settings. "The reality of AI today is that it's far easier to build and deploy systems than to evaluate and understand them," she said. She noted that the study required a team of clinical experts and more than a year of work to uncover these risks. By contrast, much of today's AI is assessed using automatic metrics that are static and lack a human in the loop. "There is a real opportunity for AI to play a role in combating the mental health crisis," Pavlick added, "but it's of the utmost importance that we take the time to really critique and evaluate our systems every step of the way to avoid doing more harm than good." The message running through the research is clear. These systems can imitate the style and language of therapy. But without reliable ethics, safety, and accountability, sounding like a therapist is not the same as being one. The findings were presented at the AAAI/ACM Conference on Artificial Intelligence, Ethics and Society. -- - Like what you read? Subscribe to our newsletter for engaging articles, exclusive content, and the latest updates.

AI Therapist? It Falls Short, a New Study Warns

By HealthDay Staff HealthDay ReporterTUESDAY, March 3, 2026 (HealthDay News) -- More people are asking artificial intelligence (AI) chatbots for help with daily problems, from work stress to relationship worries and more. Now, a new study warns that when it comes to mental health advice, these systems may fall short. A team at Brown University in Providence, Rhode Island, found that even when AI systems are told to act like trained therapists, they often fail to meet professional ethics standards. The team worked with mental health experts to examine how these systems respond in counseling-like conversations. Their study examined a series of ethical risks to show how large language model (LLM) counselors violate standards in mental health practice. Several major AI systems, including versions of OpenAI's GPT models, Anthropic's Claude and Meta's Llama, were tested. For the study, researchers asked trained peer counselors to hold practice therapy sessions with the AI, using prompts designed to make the systems act like cognitive behavioral therapy (CBT) or dialectical behavior therapy (DBT) counselors. Three licensed psychologists then reviewed the AI responses. The study identified 15 different risks, grouped into five main areas: "We call on future work to create ethical, educational and legal standards for LLM counselors -- standards that are reflective of the quality and rigor of care required for human-facilitated psychotherapy," researchers wrote. Lead researcher Zainab Iftikhar, a doctoral candidate in computer science at Brown, said prompts alone are not enough to make AI systems safe for therapy. "Prompts are instructions that are given to the model to guide its behavior," she explained. "You don't change the underlying model or provide new data, but the prompt helps guide the model's output based on its pre-existing knowledge and learned patterns. Many people share therapy-style prompts on TikTok, Instagram and Reddit. Some consumer mental health apps also use these prompt techniques to power AI chat features. Iftikhar noted that human therapists can also make mistakes. The difference, she said? Oversight. "For human therapists, there are governing boards and mechanisms for providers to be held professionally liable for mistreatment and malpractice," Iftikhar said in a news release. "But when LLM counselors make these violations, there are no established regulatory frameworks." Researchers said AI tools could still help expand access to mental health support, especially for people who cannot afford or find a licensed professional. But they stressed that stronger safeguards are needed before relying on these systems in serious situations. "If you're talking to a chatbot about mental health, these are some things that people should be looking out for," Iftikhar said. Ellie Pavlick, a computer science professor at Brown who was not involved in the study, spoke to the need for moving deliberately. "The reality of AI today is that it's far easier to build and deploy systems than to evaluate and understand them," she said. "There is a real opportunity for AI to play a role in combating the mental health crisis that our society is facing, but it's of the utmost importance that we take the time to really critique and evaluate our systems every step of the way to avoid doing more harm than good," Pavlick said. "This work offers a good example of what that can look like." The findings were presented at a conference of the Association for the Advancement of Artificial Intelligence and Association for Computing Machinery. Research presented at meetings is considered preliminary until published in a peer-reviewed journal. More information Stanford University's Institute for Human-Centered Artificial Intelligence has explored the dangers of AI in mental health care. SOURCE: Brown University, news release, March 2, 2026