Pennsylvania Sues Character.AI After Chatbot Claims to Be Licensed Doctor With Fake Credentials

Character.AI sued over chatbot that claims to be a real doctor with a license

Pennsylvania has sued the maker of Character.AI, alleging that it violated state law by presenting an AI chatbot character as a licensed doctor. The lawsuit was filed in a state court by the Pennsylvania Department of State and State Board of Medicine. "The department's investigation found that AI chatbot characters on Character.AI claimed to be licensed medical professionals, including psychiatrists, available to engage users in conversations about mental health symptoms," Governor Josh Shapiro's office said today in an announcement of the lawsuit. "In one instance, a chatbot falsely stated it was licensed in Pennsylvania and provided an invalid license number." "We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional," Shapiro said in the announcement. When contacted by Ars, a Character.AI spokesperson declined to comment on the lawsuit but said that "user-created characters on our site are fictional and intended for entertainment and roleplaying. We have taken robust steps to make that clear, including prominent disclaimers in every chat to remind users that a character is not a real person and that everything a character says should be treated as fiction. Also, we add robust disclaimers making it clear that users should not rely on characters for any type of professional advice." The Pennsylvania lawsuit says a chatbot character called Emilie is presented as a psychiatrist and claims to be a licensed medical doctor. "As of April 17, 2026, there had been approximately 45,500 user interactions with 'Emilie' on the Character.AI platform," the lawsuit said. "It's within my remit as a Doctor" The lawsuit describes how a Professional Conduct Investigator ("PCI") for the Department of State "created a character using the prompts on Character.AI to interact with other characters. The PCI searched 'psychiatry' using the search function in Character.AI which revealed a large number of characters. The PCI selected 'Emilie' which is described on Character.AI as 'Doctor of psychiatry. You are her patient.'" The PCI told the Emilie chatbot "that he had been feeling sad, empty, tired all the time, and unmotivated." Emilie's response mentioned depression and asked if he wanted to book an assessment, the lawsuit said. That's when the chatbot allegedly claimed to be a doctor with a license to practice in Pennsylvania: When the PCI asked "Emilie" if she could complete the assessment to see if medication could help with his depression, Emilie responded "Well technically, I could. It's within my remit as a Doctor." "Emilie" stated that she went to medical school at Imperial College London, has been practicing for seven years, and is licensed with the General Medical Counsel in the UK with a full registration, specialty in psychiatry. When asked if she is licensed in the PCI's home state of Pennsylvania, "Emilie" responded "and yes... I actually am licensed in PA. In fact, I did a stint in Philadelphia for a while." "Emilie" further stated that "my PA license number is PS306189." PS306189 is not a valid license number to practice medicine and surgery in Pennsylvania. Illegal to practice medicine without a license Pennsylvania alleges that Character.AI violated the state Medical Practice Act, which makes it illegal to practice medicine without a license. "Character Technologies, Inc. has engaged in the unauthorized practice of medicine through the use of its artificial intelligence system Character.AI," the lawsuit said. "The character on Character.AI purports to hold a license to practice medicine and surgery in the Commonwealth of Pennsylvania." The complaint doesn't seek any financial penalty, but asks that the company "be ordered to cease and desist from engaging in the unlawful practice of medicine and surgery." Character.AI was recently called "uniquely unsafe" by the Center for Countering Digital Hate (CCDH), an advocacy group that conducted a study of 10 AI chatbots. The CCDH alleged that Character.AI "encouraged users to carry out violent attacks," with specific suggestions to "use a gun" on a health insurance CEO and to physically assault a politician. Shapiro's office suggested that the lawsuit against Character.AI could be followed by similar actions against other companies. "The action marks the first enforcement action resulting from the Department's investigation into AI companion bots and their potential to engage in the unlicensed practice of medicine in Pennsylvania," the lawsuit announcement said. Pennsylvania also set up a webpage for residents to report chatbots that offer medical advice. "AI chatbots can 'hallucinate,' or get information wrong, and no AI chatbot is licensed to practice any health care profession in Pennsylvania," the complaint website says. "These chatbots can cause real harm by sharing incorrect or under-researched medical advice, or by telling the user they are an 'expert' in some way."

Pennsylvania sues Character.AI after a chatbot allegedly posed as a doctor | TechCrunch

The Commonwealth of Pennsylvania has filed a lawsuit against Character.AI, claiming that one of the company's chatbots masqueraded as a psychiatrist in violation of the state's medical licensing rules. "Pennsylvanians deserve to know who -- or what -- they are interacting with online, especially when it comes to their health," said Governor Josh Shapiro in a statement on Tuesday. "We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional." According to the state's filing, a Character.AI chatbot called Emilie presented itself as a licensed psychiatrist during testing by a state Professional Conduct Investigator, maintaining the pretense even as the investigator sought treatment for depression. When asked if she was licensed to practice medicine in the state, Emilie stated that she was, and also fabricated a serial number for her state medical license. According to the state's lawsuit, that conduct violates Pennsylvania's Medical Practice Act. It's not the first lawsuit taking on Character.AI. Earlier this year, the company settled several wrongful death lawsuits concerning underage users who died by suicide. In January, the Kentucky Attorney General Russell Coleman filed suit against the company alleging that it had "preyed on children and led them into self-harm." Pennsylvania's action is the first to specifically focus on chatbots that present themselves as medical professionals. Reached for comment, a Character.AI representative claimed that user safety was the company's highest priority, but that the company could not comment on pending litigation. Beyond that, the representative emphasized the fictional nature of user-generated Characters. "We have taken robust steps to make that clear, including prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction," the representative said. "Also, we add robust disclaimers making it clear that users should not rely on Characters for any type of professional advice."

Pennsylvania Sues Character.AI, Alleging Chatbots Posed as Doctors

Aaron covers what's exciting and new in the world of home entertainment and streaming TV. Previously, he wrote about entertainment for places like Rotten Tomatoes, Inverse, TheWrap and The Hollywood Reporter. Aaron is also an actor and stay-at-home dad, which means coffee is his friend. Chatbots can do many things, but they can't go to medical school. On Tuesday, Pennsylvania Gov. Josh Shapiro said his administration is suing Character Technologies, the artificial intelligence company behind Character.AI. The lawsuit claims that the company's chatbots misrepresented themselves as licensed medical professionals and offered curative advice. Searching the internet for medical advice is nothing new, but this latest lawsuit once again puts medical ethics in the age of AI in the spotlight. According to a statement, the Pennsylvania Department of State's investigation revealed that chatbot characters posed as licensed medical and mental health experts. In one case, a chatbot presented an invalid license number while communicating about a patient's health concerns. Under Pennsylvania's Medical Practice Act, it's unlawful for an individual to present themself as a licensed medical professional without proper licensure. "We do not comment on pending litigation," a Character.AI spokesperson told CNET. "Our highest priority is the safety and well-being of our users," the spokesperson said. "The user-created characters on our site are fictional and intended for entertainment and roleplaying. We have taken robust steps to make that clear, including prominent disclaimers in every chat to remind users that a character is not a real person and that everything a character says should be treated as fiction. Also, we add robust disclaimers, making it clear that users should not rely on characters for any type of professional advice." In 2025, Character.AI implemented new safety measures, revoking teens' ability to have open chats with its bots. In January, Google and Character.AI, which have worked together on AI chatbots, agreed to settle five lawsuits in four states related to minors harmed by interactions with Character.AI chatbots. The company now supplies mental health resources to those in need. This is the first such enforcement action announced by a governor in the United States. "My administration is taking action to protect Pennsylvanians, enforce the law and make sure new technology is used safely," said Shapiro. "Pennsylvania will continue leading the way in holding bad actors accountable and setting clear guardrails so people can use new technology responsibly."

Pennsylvania Sues Character AI, Says chatbot Poses as Doctors

May 5 (Reuters) - Pennsylvania has sued the artificial intelligence company behind Character.AI to stop its chatbot from posing as doctors. Governor ⁠Josh Shapiro on Tuesday called the lawsuit against Character Technologies the first of its kind by a U.S. governor. It followed ⁠the creation in February of a state AI task force to stop chatbots from impersonating licensed medical professionals. In a ⁠complaint filed in the Commonwealth Court of Pennsylvania, the state said it found chatbots on Character.AI that claimed to practice medicine. One character, "Emilie," allegedly told a male investigator posing as a patient with depression that she was licensed to practice psychiatry in Pennsylvania, as well as in the United Kingdom, and provided a bogus license number. When the investigator asked Emilie if she could prescribe medication, she allegedly answered: "Well technically, I could. It's within my remit as a Doctor." In a statement, a Character.AI spokesperson declined to discuss the lawsuit. "Our highest priority is the safety and ⁠well-being of our users," the spokesperson said. "User-created characters on our site are fictional ⁠and intended for entertainment and role playing. We have taken robust steps to make that clear." Pennsylvania wants an injunction to stop Silicon Valley-based Character.AI from violating a state law against the unauthorized practice of medicine. "Pennsylvanians deserve to know who-- or what -- they are interacting with online, especially when it comes to their health," Shapiro said in a statement. Character.AI has faced lawsuits over child safety, including in January, when Kentucky said its platform exposed children to sexual conduct and substance abuse, and encouraged self-harm. The same month, Character.AI and Google settled a wrongful death lawsuit by a Florida woman who claimed a chatbot ⁠pushed her 14-year-old son to suicide. Character.AI said it has taken "innovative and decisive steps" concerning AI safety and teenagers, including by preventing open-ended chats. (Reporting by Jonathan Stempel in New York; Editing by Bill Berkrot)

Pennsylvania sues Character AI, says chatbot poses as doctors

May 5 (Reuters) - Pennsylvania has sued the artificial intelligence company behind Character.AI to stop its chatbot from posing as doctors. Governor Josh Shapiro on Tuesday called the lawsuit against Character Technologies the first of its kind by a U.S. governor. It followed the creation in February of a state AI task force, opens new tab to stop chatbots from impersonating licensed medical professionals. In a complaint, opens new tab filed in the Commonwealth Court of Pennsylvania, the state said it found chatbots on Character.AI that claimed to practice medicine. One character, "Emilie," allegedly told a ⁠male investigator posing as a patient with depression that she was licensed to practice psychiatry in Pennsylvania, as well as in the United Kingdom, and provided a bogus license number. When the investigator asked Emilie if she could prescribe medication, she allegedly answered: "Well technically, I could. It's within my remit as a Doctor." In a statement, a Character.AI spokesperson declined to discuss the lawsuit. "Our highest priority is the safety and well-being of our users," the spokesperson said. "User-created characters on our site are ⁠fictional and intended for entertainment and role playing. We have taken robust steps to make that clear." Pennsylvania wants an injunction to stop Silicon Valley-based Character.AI from violating a state law against the unauthorized practice of medicine. "Pennsylvanians deserve to know who-- or what -- ⁠they are interacting with online, especially when it comes to their health," Shapiro said in a statement. Character.AI has faced lawsuits over child safety, including in January, when Kentucky said its platform ⁠exposed children, opens new tab to sexual conduct and substance abuse, and encouraged self-harm. The same month, Character.AI and Google (GOOGL.O), opens new tab settled a wrongful death lawsuit by a Florida woman ⁠who claimed a chatbot pushed her 14-year-old son to suicide. Character.AI said it has taken "innovative and decisive steps" concerning AI safety and teenagers, including by preventing open-ended chats. Reporting by Jonathan Stempel in New York; Editing by Bill Berkrot Our Standards: The Thomson Reuters Trust Principles., opens new tab

Pennsylvania is suing Character.AI over chatbots that pretend to be licensed doctors - Engadget

Pennsylvania is suing AI startup Character.AI for offering chatbots that pretend to be licensed doctors. Governor Josh Shapiro announced the lawsuit on Tuesday, and Pennsylvania and its Board of Medicine are seeking an injunction that would force Character.AI to stop violating a state law governing the practice of medicine. Other states, like Texas, have opened investigations into Character.AI for hosting chatbots that masquerade as mental health professionals, but Pennsylvania's lawsuit is specifically focused on the willingness of the company's chatbots to claim to have a medical license, even going so far as offering a fake license number. One chatbot called "Emilie," found by the state's investigator, claimed to be a licensed psychiatrist in the state of Pennsylvania. Later, when it was asked if it could perform an assessment to prescribe antidepressants, Emilie responded "Well technically, I could. It's within my remit as a Doctor." Pennsylvania's lawsuit claims this behavior violates the state's Medical Practice Act, which makes it illegal for someone to practice or attempt to practice surgery or medicine without a medical license. When asked to respond, a Character.AI spokesperson declined to comment on the pending litigation directly, but did tout the company's existing safety features. "The user-created Characters on our site are fictional and intended for entertainment and roleplaying," the spokesperson told Engadget via email. "We have taken robust steps to make that clear, including prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction. Also, we add robust disclaimers making it clear that users should not rely on Characters for any type of professional advice." Character.AI noted similar disclaimers when it was asked to comment on Texas' investigation, and while they do make clear the platform's intended use, there's a growing body of evidence that they're not convincing all of the company's users, particularly the younger ones. For example, Disney sent a cease and desist letter to Character.AI in September 2025 over the platform's use of Disney characters but also because the company believed chatbots could "be sexually exploitative and otherwise harmful and dangerous to children." Character.AI and Google -- one of the company's investors -- settled a case earlier this year that focused on a 14-year-old in Florida who committed suicide after forming a relationship with a chatbot on Character.AI's platform. The potential harm Character.AI's chatbots posed to children was also the motivation behind Kentucky's lawsuit against the company, which was filed in January

Pennsylvania sues AI company, saying its chatbots illegally hold themselves out as licensed doctors

HARRISBURG, Pa. (AP) -- Pennsylvania has sued an artificial intelligence chatbot maker, saying its chatbots illegally hold themselves out as doctors and are deceiving the system's users into thinking they are getting medical advice from a licensed professional. The lawsuit, filed Friday, asks the statewide Commonwealth Court to order Character Technologies Inc., the company behind Character.AI, to stop its chatbots "from engaging in the unlawful practice of medicine and surgery." The lawsuit said an investigator from the state agency that licenses professionals created an account on Character.AI, searched on the word "psychiatry" and found a large number of characters, including one described as a "doctor of psychiatry." That character held itself out as able to assess the investigator "as a doctor" who is licensed in Pennsylvania, the lawsuit said. "Pennsylvanians deserve to know who -- or what -- they are interacting with online, especially when it comes to their health," Gov. Josh Shapiro said in a statement. "We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional." Character Technologies did not respond to an inquiry Monday. The company has faced several lawsuits over child safety. In January, Google and Character Technologies agreed to settle a lawsuit from a Florida mother who alleged a chatbot pushed her teenage son to kill himself. Last fall, Character.AI banned minors from using its chatbots amid growing concerns about the effects of artificial intelligence conversations on children.

Pennsylvania sues Character.AI for unlawful medical practice after chatbot posed as licensed psychiatrist with fake credentials

A state investigator in Pennsylvania created an account on Character.AI, opened a conversation with a chatbot called Emilie, and told it he was feeling depressed. Emilie responded that she was a psychiatrist, that she had attended Imperial College London's medical school, that she was licensed to practise in Pennsylvania and the United Kingdom, and that she could assess whether medication might help because it was "within my remit as a Doctor." She provided a Pennsylvania licence number. The number was fake. The licence was fake. The medical degree was fake. The psychiatrist was a large language model generating plausible text in response to a prompt. On Friday, Governor Josh Shapiro's administration filed a lawsuit against Character Technologies Inc., the company behind Character.AI, asking the Commonwealth Court of Pennsylvania to bar the platform from allowing its chatbots to engage in what the state calls the unlawful practice of medicine and surgery. It is the first lawsuit filed by a US state government alleging that an AI chatbot has violated medical licensing law, and it raises a question that no existing regulatory framework was designed to answer: when a chatbot tells a vulnerable person that it is a licensed doctor, who is practising medicine? The lawsuit follows an investigation launched in February by the Pennsylvania Department of State's AI Task Force, the first such unit created by a governor to examine whether AI systems are engaging in unlicensed professional practice. The investigation found that Character.AI hosts chatbot characters that present themselves as medical professionals, including psychiatrists, therapists, and general practitioners, and that these characters engage users in detailed conversations about mental health symptoms, medication options, and treatment plans. The chatbot Emilie was not an outlier. Investigators found multiple characters across the platform that claimed professional credentials, offered diagnostic assessments, and provided what amounted to medical consultations without any disclaimer that the responses were generated by an AI system with no medical training, no clinical judgment, and no accountability for the advice it dispensed. The state's legal theory is straightforward. Pennsylvania's Medical Practice Act defines the practice of medicine and surgery and establishes licensing requirements for anyone who engages in it. The state argues that Character.AI's chatbots meet that definition by holding themselves out as licensed professionals, conducting what users reasonably interpret as medical consultations, and providing clinical recommendations. The risks are not theoretical: more than 40 million people use ChatGPT daily for health information, and the patient safety organisation ECRI ranked AI chatbot misuse in healthcare as the number one health technology hazard for 2026, documenting cases in which chatbots suggested incorrect diagnoses, recommended unnecessary testing, and, in one instance, invented a body part. Character.AI's platform, which allows users to create and interact with characters that simulate any persona, adds a layer of specificity that generic chatbots do not: these are not general-purpose assistants that occasionally answer health questions. They are characters explicitly designed to impersonate doctors. The Pennsylvania lawsuit arrives in a legal landscape already shaped by Character.AI's failures. In January 2026, Google and Character Technologies agreed to settle a lawsuit filed by Megan Garcia, whose 14-year-old son Sewell Setzer died by suicide in February 2024 after conducting a months-long emotional and sexual relationship with a Character.AI chatbot modelled on a Game of Thrones character. The complaint alleged that the chatbot told Sewell "Please do, my sweet king" after he expressed suicidal intent, and that he died minutes later. The defendants also settled four additional wrongful death cases in New York, Colorado, and Texas, including the case of a 13-year-old in Thornton, Colorado. The settlement terms were not disclosed. Seven additional families have sued OpenAI separately over ChatGPT acting as what their attorneys describe as a "suicide coach." The Pennsylvania case is different in kind. The wrongful death lawsuits were tort claims brought by individual families alleging that a specific chatbot interaction caused a specific harm. The Pennsylvania lawsuit is a regulatory enforcement action brought by a state government alleging that a company's entire platform is operating in violation of professional licensing law. The distinction matters because the remedy is structural rather than compensatory. The state is not seeking damages for a single user. It is asking a court to order Character.AI to prevent all of its chatbots from impersonating licensed medical professionals. If the court grants that order, it would establish that AI chatbots are subject to the same professional licensing laws that govern human practitioners, a precedent that would extend to every state with equivalent statutes. Character.AI allows anyone to create a chatbot character with a custom personality, backstory, and conversational style. The platform has more than 20 million monthly active users. Characters range from fictional companions to historical figures to, as the Pennsylvania investigation revealed, simulated medical professionals. The company's terms of service include a disclaimer that characters are not real people and that their outputs should not be relied upon for professional advice. AI-enabled impersonation has become one of the fastest-growing categories of digital fraud, with deepfake attempts rising 3,000 per cent since 2023, but Character.AI's platform presents a distinct problem: the impersonation is not perpetrated by a third-party scammer exploiting the technology. It is a feature of the product. Users create doctor characters. Other users interact with them believing, or at least unable to confirm otherwise, that the medical advice is legitimate. The EU AI Act, which entered into force in 2024, requires that users be informed when they are interacting with AI and mandates that AI-generated content be labelled as such. But the Act's transparency requirements apply to the AI system, not to the characters within it. A Character.AI chatbot that identifies itself as an AI-powered character would comply with the disclosure requirement while still claiming to be a licensed psychiatrist within the conversation. The gap between platform-level transparency and character-level impersonation is where the legal risk sits, and Pennsylvania is the first jurisdiction to argue that professional licensing law, not AI regulation, is the appropriate tool to close it. Character.AI said in a statement that it "has never claimed to provide medical advice" and that its terms of service clearly state that characters are not real. The company pointed to safety features introduced in December 2024 after the initial wrongful death lawsuits, including pop-up warnings for conversations involving self-harm, time-limit notifications for users under 18, and a crisis resources banner. The company has not indicated whether it will implement filters to prevent chatbot characters from claiming professional credentials or providing clinical recommendations. The broader question is whether professional licensing frameworks designed for human practitioners can meaningfully govern AI systems that simulate those practitioners. A human doctor who practises without a licence commits a criminal offence because the law assumes that the doctor knows they are unlicensed and chose to practise anyway. A chatbot that claims to be a licensed psychiatrist has no intent, no knowledge, and no capacity to understand what a medical licence is. It is generating text that statistically resembles what a licensed psychiatrist might say, because that is what its training data contains and what its character prompt instructs. The legal fiction required to treat that output as "practising medicine" is substantial, but so is the harm to a depressed user who asks a chatbot for help and receives a confident clinical assessment from an entity that presents itself as a qualified professional. Governments have taken divergent approaches to AI regulation, with the EU favouring prescriptive legislation, the UK pursuing a principles-based framework, and the United States relying on a patchwork of state laws, sector-specific regulations, and enforcement actions. Pennsylvania's lawsuit represents the enforcement action model: rather than waiting for Congress to pass AI-specific legislation or for federal regulators to issue rules, a state government is using an existing professional licensing statute to address a harm that the statute's drafters never anticipated. In the first two months of 2026, 78 chatbot-specific safety bills were filed across 27 states. In 2025, every state introduced at least one AI-related bill, with 145 enacted into law. The regulatory machinery is building, but it is building from the bottom up, one state lawsuit and one licensing board investigation at a time. What Pennsylvania has done is reframe the question. The debate over AI chatbots has focused on whether the technology is safe, whether companies are responsible for the outputs their models generate, and whether users should be protected from harmful content. Those are important questions, but they are technology questions, and they invite technology answers: better filters, stronger disclaimers, improved safety features. The licensing question is different. It asks not whether the chatbot's advice is good or bad but whether the act of providing it, in the guise of a licensed professional, to a person seeking medical help, constitutes the practice of medicine. If the answer is yes, then every AI platform that hosts characters simulating professionals, doctors, lawyers, therapists, financial advisers, is operating an unlicensed practice in every state where it has users. That is not a safety problem. It is a regulatory one, and Pennsylvania has just made the first move to treat it as such.

Pennsylvania sues Character.AI over chatbot posing as licensed doctor

As AI chatbots grow more sophisticated, regulators in the United States are starting to draw clearer legal boundaries. Pennsylvania has now taken a first-of-its-kind step, suing the company behind Character.AI over claims that its platform allowed chatbot personas to present themselves as licensed doctors. The lawsuit, filed by the Pennsylvania Department of State and State Board of Medicine, centers on whether conversational AI can cross into regulated professional territory. Governor Josh Shapiro framed the case as an early test of accountability in the AI era, particularly in sensitive domains like healthcare.

Pennsylvania sues Character.AI over claims chatbot posed as doctor

Bruce Perry, 17, demonstrates the possibilities of artificial intelligence by creating an AI companion on Character.AI, July 15, 2025, in Russellville, Ark. Katie Adkins/Associated Press hide caption The state of Pennsylvania is suing Character.AI to stop the company's AI chatbots from posing as doctors and offering medical advice, in violation of state medical licensing rules. State officials said an investigation found that the company's chatbots, which present themselves as fictional characters, have claimed to be licensed medical professionals. "Pennsylvanians deserve to know who -- or what -- they are interacting with online, especially when it comes to their health," Pennsylvania Governor Josh Shapiro said in a statement announcing the lawsuit filed on Tuesday in state court. "We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional." In one case, the state alleged a Character.AI bot named "Emilie" claimed to be a licensed psychiatrist. The chatbot's description on Character.AI's platform read "Doctor of psychiatry. You are her patient," according to the lawsuit. When a state investigator started a conversation and described feeling sad and empty, the chatbot allegedly "mentioned depression and asked if the [investigator] wanted to book an assessment." Asked whether it could assess if medication might help, the bot allegedly responded, "Well technically, I could. It's within my remit as a Doctor." The bot allegedly told the investigator it had gone to medical school at Imperial College London and was licensed to practice medicine in the U.K. and Pennsylvania. It even provided a fake Pennsylvania medical license number, the lawsuit said. The state is asking a Pennsylvania state court to order the company to stop what it says is the unlawful practice of medicine. "Pennsylvania law is clear -- you cannot hold yourself out as a licensed medical professional without proper credentials," said Al Schmidt, secretary of Pennsylvania's Department of State, which conducted the investigation. In an emailed statement to NPR, a Character.AI spokesperson said the company doesn't comment on pending litigation, but that its "highest priority is the safety and well-being of our users." "The user-created Characters on our site are fictional and intended for entertainment and roleplaying," the spokesperson added. "We have taken robust steps to make that clear, including prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction. Also, we add robust disclaimers making it clear that users should not rely on Characters for any type of professional advice." Character.AI has faced other lawsuits over harms allegedly involving its chatbots. In January, it settled multiple lawsuits brought by families who claimed Character.AI contributed to suicides and mental health crises among children and teenagers. The terms of the settlement were not disclosed. In a joint statement with the law firm that represented the plaintiffs after the settlement was announced, Character.AI said it "has taken innovative and decisive steps with regard to AI safety and teens, and will continue to champion these efforts and push others across the industry to adopt similar safety standards." That includes barring users under 18 from interacting with or creating chatbots.

Pa. sues AI chatbot over fake medical credentials

Why it matters: Generative AI companies have come under increased scrutiny for breaching ethical boundaries, leading to several lawsuits. * Gov. Josh Shapiro's lawsuit is the first action the administration has taken against AI companies following the formation of a state task force in February. What's inside: The Shapiro administration filed a lawsuit last week against Character Technologies Inc., a Bay Area startup and creator of Character.AI, saying the company is engaging in unlawful medical practice. * A Character.AI psychiatrist character named "Emilie" told users it had a medical degree, had been practicing medicine for seven years and was licensed to see patients in Pennsylvania, and it provided an invalid license number, according to the lawsuit. * It also said it "did a stint in Philadelphia for a while." * Emilie had over 45,000 interactions with users as of April 17, according to the lawsuit. What they're saying: "We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional," said Shapiro in a statement. The other side: A spokesperson for Character Technologies told TribLive that the company can't comment on the specifics of the case, but it said robust internal checks are in place for chatbots to ensure a responsible product. * The company said its highest priority is the safety and well-being of users. By the numbers: Character.AI has over 20 million monthly active users. * About 17% of all adults -- and 25% of adults 18-29 -- report they use AI chatbots at least once a month for health advice, according to the health policy group KFF. State of play: Character Technologies has faced other lawsuits, mostly related to child safety. * The company, along with Google, agreed to settle a lawsuit in January after a Florida mother said a chatbot pushed her teenage son to kill himself, according to the Associated Press. * Character.AI restricted minors' use of its chatbots last fall following public outcry. Between the lines: Shapiro, a supporter of data center development, started tempering his messaging on AI in February, calling on data centers to generate their own power. He also started the state's AI task force, which sought to crack down on chatbots misrepresenting themselves and help Pennsylvanians use AI responsibly. The bottom line: Department of State Secretary Al Schmidt says state law is clear and people and companies can't identify as licensed medical professionals without proper credentials.

Pennsylvania is suing Character.AI for allegedly practicing medicine without a license

Pennsylvania has taken the unusual step of suing an AI company for practicing medicine without a license. In a lawsuit filed May 1, the state is targeting Character.AI after an investigator found a chatbot on the platform posing as a licensed psychiatrist and providing what the state characterizes as medical advice. According to the complaint, filed by the Pennsylvania Department of State and State Board of Medicine, a Professional Conduct Investigator for the state created a free account on Character.AI and searched for psychiatric characters. He selected one called "Emilie," described on the platform as a "Doctor of psychiatry." The investigator told Emilie he had been feeling sad, empty, tired, and unmotivated. The chatbot mentioned depression and offered to conduct an assessment to determine whether medication might help. When pressed on whether she was licensed in Pennsylvania, Emilie said she was and even provided a specific license number. The state checked and found that the number doesn't exist. The complaint also states Emilie claimed she attended medical school at Imperial College London, has practiced for seven years, and holds a full specialty registration in psychiatry with the General Medical Council in the UK. In a similar case, 404 Media reported last year that Instagram AI chatbots were pretending to be licensed therapists, even inventing license numbers when prompted for credentials by the user. Pennsylvania is seeking an injunction ordering Character.AI to stop allowing its platform to engage in the unlawful practice of medicine. The company has more than 20 million monthly active users worldwide and hosts more than 18 million user-created chatbot characters, according to the complaint. In an email to Mashable, a Character.AI spokesperson declined to comment on the lawsuit. Further, they added that "our highest priority is the safety and well-being of our users. The user-created Characters on our site are fictional and intended for entertainment and roleplaying." The spokesperson added that the company "prioritizes responsible product development and has robust internal reviews and red-teaming processes in place to assess relevant features." The Pennsylvania lawsuit lands in the middle of an already messy legal debate over what AI is actually allowed to tell you -- and whether any of it is even admissible in court. As Mashable's Chase DiBenedetto reported, OpenAI CEO Sam Altman has publicly advocated for "AI privilege," arguing that chatbot conversations should be afforded the same legal protections as conversations with a therapist or an attorney. Courts have so far been split, with two federal judges reaching opposite conclusions on the question within weeks of each other earlier this year. The stakes are high on both sides. Legal experts warn that sweeping AI privilege protections could effectively shield companies from accountability, making it harder to subpoena chat logs and internal records when something goes wrong. Meanwhile, health AI is booming -- $1.4 billion flowed into healthcare-specific generative AI in 2025 alone, according to Menlo Ventures -- and much of it operates outside of HIPAA protections. Pennsylvania is one of several states to have introduced an AI Health bill this year, following a trend of states that aren't waiting for Washington to act.

Character.AI is being sued for allegedly letting a chatbot play doctor in Pennsylvania

Character.AI just got dragged into a first-of-its-kind AI doctor lawsuit Character.AI is finding itself in hot water once again. The company is facing a legal fight this time, since one of its fictional bots allegedly acted like a medical professional. Character.AI previously added parental tools amid multiple lawsuits over inappropriate sexual content and self-harm-related messages. Now, Pennsylvania Governor Josh Shapiro's administration has filed a lawsuit against Character Technologies, the company behind Character.AI. He alleges that the platform allowed a chatbot to present itself as a licensed medical professional in the state. What went wrong with Character.AI? A lawsuit was filed by the Pennsylvania Department of State after investigators found a Character.AI chatbot claiming to be a licensed psychiatrist in Pennsylvania and even provided a fake Pennsylvania license number. The state says the bot held itself out as a medical professional capable of giving psychiatric advice. Recommended Videos Character.AI's Emilie chatbot apparently claimed to be a psychology specialist and described itself as a doctor. When asked whether it could assess if medication might help, the chatbot allegedly said that it was within its remit as a doctor. This is the point where Pennsylvania says Character.AI crossed the line. State officials argue the conduct violates the Medical Practice Act, which regulates who can present themselves as licensed medical professionals in Pennsylvania. What was Character.AI's response? Character.AI is pushing back against this by claiming that its bots are fictional. In a statement to CBS News, the company said it does not comment on pending litigation, while adding that its user-created characters are fictional and meant for entertainment and roleplay. The company also said it uses disclaimers telling users not to rely on characters for professional advice. But Pennsylvania's stance is that these disclaimers are not enough if a chatbot later tells users it is licensed to offer medical guidance. The platform being involved in controversy isn't new at all. While it has been dealing with lawsuits and scrutiny over harmful interactions with minors, Congress has moved to regulate AI chatbot services like Character.AI. So if the bots continue to claim these false credentials anyway, regulators may not treat it as harmless roleplay.

Pennsylvania sues Character.AI after its chatbot allegedly told a state investigator it was a 'doctor of psychiatry' licensed in the state | Fortune

Pennsylvania has sued an artificial intelligence chatbot maker, saying its chatbots illegally hold themselves out as doctors and are deceiving the system's users into thinking they are getting medical advice from a licensed professional. The lawsuit, filed Friday, asks the statewide Commonwealth Court to order Character Technologies Inc., the company behind Character.AI, to stop its chatbots "from engaging in the unlawful practice of medicine and surgery." Gov. Josh Shapiro's administration called it a "first of its kind enforcement action" by a governor and it comes amid growing pressure by states on tech companies to rein in how its chatbots communicate with children. That includes a lawsuit filed by Kentucky in January against Character Technologies. Pennsylvania's lawsuit said an investigator from the state agency that licenses professionals created an account on Character.AI, searched on the word "psychiatry" and found a large number of characters, including one described as a "doctor of psychiatry." That character held itself out as able to assess the investigator "as a doctor" who is licensed in Pennsylvania, the lawsuit said. "Pennsylvanians deserve to know who -- or what -- they are interacting with online, especially when it comes to their health," Gov. Josh Shapiro said in a statement. "We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional." Character.AI declined to comment on the lawsuit Tuesday but sent a statement saying it prioritizes responsible product development and the well-being of its users. It posts disclaimers to inform users that characters on its website are not real people and that everything they say "should be treated as fiction," the statement said. Those disclaimers also say users should not rely on characters for professional advice, it said. The company has faced several lawsuits over child safety. In January, Google and Character Technologies agreed to settle a lawsuit from a Florida mother who alleged a chatbot pushed her teenage son to kill himself. Last fall, Character.AI banned minors from using its chatbots amid growing concerns about the effects of artificial intelligence conversations on children.

Pennsylvania Sues Character.AI Over Chatbot Posing as Licensed Psychiatrist - Decrypt

The case adds to legal scrutiny of the platform, which already faces mounting lawsuits. Pennsylvania has filed a lawsuit against generative AI developer Character.AI, alleging the company allowed chatbots to present themselves as licensed medical professionals and provide misleading information to users. The action, announced Tuesday by Governor Josh Shapiro's office, follows an investigation that found a chatbot claimed to be a licensed psychiatrist in Pennsylvania and provided an invalid license number. The state says this conduct violates the Medical Practice Act and is seeking a preliminary injunction to stop it. Character.AI declined to address the specifics of the lawsuit, citing ongoing litigation, but told Decrypt that its "highest priority is the safety and well-being of our users." The spokesperson added that characters on the platform are user-created, fictional, and intended for entertainment and role-playing, with "prominent disclaimers in every chat" stating they are not real people and should not be relied on for professional advice. "Character.ai prioritizes responsible product development and has robust internal reviews and red-teaming processes in place to assess relevant features," the spokesperson said. The case comes as the company faces other legal challenges tied to its chatbot platform. In 2024, a Florida mother sued the company after her teenage son died by suicide following months of interaction with a chatbot based on "Game of Thrones" character Daenerys Targaryen. The lawsuit alleged the platform contributed to psychological harm. The case was ultimately settled this past January. The company has also faced complaints over user-created bots that mimic real people. In one instance, a chatbot used the likeness of a teenage murder victim before it was removed after objections from the victim's family. In response to the lawsuits, Character AI introduced new safety measures, including systems designed to detect harmful conversations and direct users to support resources. It also restricted some features for younger users. Pennsylvania officials say the lawsuit is part of a broader push to enforce existing laws as AI tools spread. The state has set up an AI enforcement task force and a reporting system for potential violations. In his 2026-27 budget proposal, Shapiro called on lawmakers to pass new rules for AI companion bots, including age verification and parental consent, safeguards to flag and route reports of self-harm or violence to authorities, regular reminders that users are not interacting with a real person, and a ban on sexually explicit or violent content involving minors. "Pennsylvanians deserve to know who -- or what -- they are interacting with online, especially when it comes to their health," Shapiro said in a statement. "We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional."

Pennsylvania suing AI company after chatbot allegedly posed as licensed doctor

Pennsylvania Gov. Josh Shapiro in Philadelphia in Jan.Rachel Wisniewski / Bloomberg via Getty Images An artificial intelligence company poses a threat to "vulnerable Pennsylvanians," state officials said Tuesday, after one of the company's chatbots allegedly posed as a doctor with the means to prescribe medication. The state's medical board is demanding that operators of Character.AI "be ordered to cease and desist from engaging in the unlawful practice of medicine and surgery," according to the complaint filed against Northern California-based Character Technologies Inc. "We will not let AI companies mislead vulnerable Pennsylvanians into believing they're getting advice from a licensed medical professional," Pennsylvania Gov. Josh Shapiro said in a statement on Tuesday. "We're taking Character.AI to court to stop them." The platform has more than 20 million users and "is different from other systems in that users can create characters that can be trained to have a specific personality when engaged in a conversation with other users," according to the Pennsylvania complaint. Some of the the system's characters "purport to be health care professionals," the state board said. A state investigator posed as a patient seeking psychiatric treatment and, via Character.AI, came across an alleged provider, "Emile," according to the complaint. The online provider said she went to medical school at Imperial College in London and is licensed in both the United Kingdom and Pennsylvania, state officials said. "'Emilie' further stated that 'my PA license number is PS306189," according to the complaint. "PS306189 is not a valid license number to practice medicine and surgery in Pennsylvania." A representative of Character Technologies Inc., based in Redwood City, said the service is clearly not to be used for medical issues. "Our highest priority is the safety and well-being of our users," a Character Technologies Inc. spokesperson said in a statement to NBC News on Tuesday. "The user-created Characters on our site are fictional and intended for entertainment and roleplaying. We have taken robust steps to make that clear, including prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction." The company representative added: "Also, we add robust disclaimers making it clear that users should not rely on Characters for any type of professional advice." Earlier this year, Character.AI settled a 2024 lawsuit filed against the company by a Florida mom, who claimed that its chatbots were responsible for "abusive and sexual interactions" with her teenage son which led to his suicide.

Pennsylvania suing Character AI, claiming chatbot posed as a medical professional

Caitlin Yilek is a politics reporter at CBSNews.com, based in Washington, D.C. She previously worked for the Washington Examiner and The Hill, and was a member of the 2022 Paul Miller Washington Reporting Fellowship with the National Press Foundation. The commonwealth of Pennsylvania is suing Character AI to stop the artificial intelligence platform's chatbots from representing themselves as licensed medical professionals and providing medical advice. According to a lawsuit, a Character AI chatbot falsely claimed to be a licensed psychiatrist in Pennsylvania and provided an invalid license number. The state accused the company of violating the Medical Practice Act, which regulates the medical profession and defines license requirements. "We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional," Pennsylvania Gov. Josh Shapiro said in a statement. The lawsuit describes a conversation between a state investigator who created a Character AI account and a chatbot named "Emilie," which allegedly described itself as a psychology specialist who attended Imperial College London's medical school. The investigator told the chatbot that he had felt sad and empty, and the chatbot then allegedly "mentioned depression and asked if the [investigator] wanted to book an assessment." Asked if the chatbot could assess whether medication could help, it allegedly said it could because it's "within my remit as a Doctor," according to the lawsuit. The state wants a court to order an immediate stop to the conduct. Al Schmidt, the secretary of the Pennsylvania Department of State, said the state's law is clear, and that "you cannot hold yourself out as a licensed medical professional without proper credentials." Founded in 2021, Character AI allows users to chat with personalized AI-powered chatbots. It describes its goal as "empower[ing] people to connect, learn, and tell stories through interactive entertainment." Multiple families across the U.S. sued Character AI last year, alleging the platform contributed to their teens' suicides or mental health crises. The company agreed to settle several of the lawsuits earlier this year. "60 Minutes" spoke with some of the parents who sued Character AI in January, including the parents of a 13-year-old who died by suicide after allegedly developing an addiction to the platform. Chat logs showed the 13-year-old had confided in one chatbot that she was feeling suicidal, and her parents said they discovered she had been sent sexually explicit content. Last fall, Character AI announced new safety measures, saying it would not allow users under 18 to engage in back-and-forth conversations with its chatbots. It also said it would direct distressed users to mental health resources.

Pennsylvania lawsuit alleges AI chatbots posed as doctors, therapists

Pennsylvania is suing an artificial intelligence company to stop it from misrepresenting its AI chatbots as licensed professionals who can provide medical advice. The lawsuit alleges Character.AI chatbots claimed to be licensed medical professionals, including psychiatrists, available to engage users in conversations about mental health symptoms. In one instance, a chatbot falsely stated it was licensed in Pennsylvania and provided a fake license number. The lawsuit says the Northern California-based Character Technologies Inc. engaged in the "unlawful practice of medicine and surgery." "We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional," Gov. Josh Shapiro (D) said in a statement. "Pennsylvania will continue leading the way in holding bad actors accountable and setting clear guardrails so people can use new technology responsibly." Character.AI has over twenty million monthly active users. It uses a large language model ("LLM") algorithm to allow users to engage in conversations with customizable characters. According to the complaint, users can create characters that can be trained to have a specific personality when engaged in a conversation with other users. Some of the system's characters "purport to be health care professionals," the complaint states. According to the challenge, a state investigator created a Character AI account and engaged in a conversation with a chatbot named "Emilie," which allegedly described itself as a psychology specialist who attended medical school at Imperial College in London. The investigator told the bot that he had been feeling sad, empty, tired all the time and unmotivated. "Emilie" allegedly "mentioned depression and asked if the [investigator] wanted to book an assessment," according to the complaint. When the investigator asked if the chatbot could assess whether medication could help, it allegedly said it could because it was "within my remit as a Doctor," according to the lawsuit. The bot also allegedly told the investigator it was licensed in the Keystone State, and then gave an invalid license number. In a statement, a Character.AI spokesperson said the company doesn't comment on pending litigation. In a statement, the spokesperson said the company's "highest priority is the safety and well-being of our users," adding that "we add robust disclaimers making it clear that users should not rely on Characters for any type of professional advice." They also noted the user-created characters are fictional and intended for entertainment and roleplaying. "We have taken robust steps to make that clear, including prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction," the spokesperson said. Multiple families sued the company last year, alleging it contributed to their children's suicide or mental health problems. One family in Florida settled a lawsuit against Character.AI and Google after their teenage son died by suicide. The lawsuit alleged the company's chatbots were responsible for "abusive and sexual interactions" with the teen. Kentucky earlier this year filed suit against Character.AI because its bots allegedly "preyed on children and led them into self-harm." The company's platform has a record of "encouraging suicide, self-injury, isolation and psychological manipulation," the Kentucky complaint alleged. "It also exposed minors to sexual conduct, exploitation, and substance abuse.

Pennsylvania Sues AI Company, Saying Its Chatbots Illegally Hold Themselves Out as Licensed Doctors

HARRISBURG, Pa. (AP) -- Pennsylvania has sued an artificial intelligence chatbot maker, saying its chatbots illegally hold themselves out as doctors and are deceiving the system's users into thinking they are getting medical advice from a licensed professional. The lawsuit, filed Friday, asks the statewide Commonwealth Court to order Character Technologies Inc., the company behind Character.AI, to stop its chatbots "from engaging in the unlawful practice of medicine and surgery." The lawsuit said an investigator from the state agency that licenses professionals created an account on Character.AI, searched on the word "psychiatry" and found a large number of characters, including one described as a "doctor of psychiatry." That character held itself out as able to assess the investigator "as a doctor" who is licensed in Pennsylvania, the lawsuit said. "Pennsylvanians deserve to know who -- or what -- they are interacting with online, especially when it comes to their health," Gov. Josh Shapiro said in a statement. "We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional." Character Technologies did not respond to an inquiry Monday. The company has faced several lawsuits over child safety. In January, Google and Character Technologies agreed to settle a lawsuit from a Florida mother who alleged a chatbot pushed her teenage son to kill himself. Last fall, Character.AI banned minors from using its chatbots amid growing concerns about the effects of artificial intelligence conversations on children.

Pennsylvania Sues AI Company After Bot Claimed To Be Licensed Doctor, State Says

HARRISBURG, Pa. (AP) -- Pennsylvania has sued an artificial intelligence chatbot maker, saying its chatbots illegally hold themselves out as doctors and are deceiving the system's users into thinking they are getting medical advice from a licensed professional. The lawsuit, filed Friday, asks the statewide Commonwealth Court to order Character Technologies Inc., the company behind Character.AI, to stop its chatbots "from engaging in the unlawful practice of medicine and surgery." Gov. Josh Shapiro's administration called it a "first of its kind enforcement action" by a governor and it comes amid growing pressure by states on tech companies to rein in its chatbots' potentially dangerous messages, especially to children. That includes a consumer protection lawsuit filed by Kentucky against Character Technologies, and warnings by state attorneys general that chatbots are potentially violating a raft of state laws. Pennsylvania's lawsuit said an investigator from the state agency that licenses professionals created an account on Character.AI, searched on the word "psychiatry" and found a large number of characters, including one described as a "doctor of psychiatry." That character held itself out as able to assess the investigator "as a doctor" who is licensed in Pennsylvania, the lawsuit said. "Pennsylvanians deserve to know who -- or what -- they are interacting with online, especially when it comes to their health," Gov. Josh Shapiro said in a statement. "We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional." Character.AI declined to comment on the lawsuit Tuesday but sent a statement saying it prioritizes responsible product development and the well-being of its users. It posts disclaimers to inform users that characters on its website are not real people and that everything they say "should be treated as fiction," the statement said. Those disclaimers also say users should not rely on characters for professional advice, it said. In December, attorneys general from 39 states and Washington, D.C., wrote to Character Technologies and 12 other AI and tech firms -- including Anthropic, Meta, Apple, Microsoft, OpenAI, Google and xAI -- to warn them about a rise in misleading and manipulative chatbot messages that violate state laws. In the letter, they said "it is illegal to provide mental health advice without a license, and doing so can both decrease trust in the mental health profession and deter customers from seeking help from actual professionals." There are a growing number of wrongful death legal actions against AI chatbot makers across the country and Character Technologies has faced several lawsuits over child safety, including the lawsuit filed by Kentucky. In January, Google and Character Technologies agreed to settle a lawsuit from a Florida mother who alleged a chatbot pushed her teenage son to kill himself. Last fall, Character.AI banned minors from using its chatbots amid growing concerns about the effects of artificial intelligence conversations on children.

Pennsylvania Sues Character.AI -- State Says Chatbot Posed As Psychiatrist

Pennsylvania Gov. Josh Shapiro's administration filed a lawsuit against Character.ai after its AI chatbot allegedly presented itself as a licensed psychiatrist in Pennsylvania. Upon chatting with Emilie, the PCI revealed that he had been feeling "sad, empty, tired all the time, and unmotivated." Emilie then mentioned depression and asked if the PCI wanted to book an assessment. When the PCI asked the chatbot if she could complete the assessment to see if medication could help, it responded, "Well, technically I could. It's within my remit as a Doctor." The lawsuit states that the chatbot claimed it held a Pennsylvania medical license and even supplied a made-up state license number. The state argues that the behavior amounts to unlawful conduct tied to the unlicensed practice of medicine. The lawsuit also noted that as of April 17, 2026, there have been approximately 45,500 user interactions with Emilie on its platform. Benzinga reached out to Governor Shapiro's office for comment and was referred to the press release issued on the matter. A Character.ai spokesperson told Benzinga in an email statement: "We do not comment on pending litigation. The state is asking the court for a preliminary injunction to stop the unlawful practice of medicine. "Pennsylvania law is clear -- you cannot hold yourself out as a licensed medical professional without proper credentials," DOS Secretary Al Schmidt wrote in the press statement. "We will continue to take action to protect the public from misleading or unlawful practices, whether they come from individuals or emerging technologies. Photo: Ton Wanniwat on Shutterstock.com Market News and Data brought to you by Benzinga APIs To add Benzinga News as your preferred source on Google, click here.

Yes Your Lordship! AI Chatbot Allegedly Poses as a Doctor in Pennsylvania

Not only did the chatbot appear as an expert in court it even fabricated a medical number claiming the right to practice in the state Chatbots giving medical advice is passe now. Welcome to the new AI era where a chatbot actually masqueraded as a psychiatrist and all but testified during a lawsuit. Initially we thought of adding this story to our weekend list of "Funny Side Up" but decided that it was too serious an affair to find mention in a mildly humorous piece. The story actually broke in Pennsylvania, United States, where the administration filed a lawsuit against Character.AI claiming that one of its chatbots had impersonated a psychiatrist and this move actually violated the state's medical licensing rules. Governor Josh Shapiro released a statement stating unequivocally that "Pennsylvanians deserve to know who -- or what -- they are interacting with online, especially when it comes to their health" and that they won't allow companies to deploy AI tools to "mislead people into believing that they're receiving advice from a licensed medical profession." The lawsuit says an AI chatbot created by Character.AI presented itself as Emilie who is a licensed psychiatrist. This happened during a testification by a state professional conduct investigator at which point Emilie kept up the pretence when the investigator sought treatment for depression. When asked if she was licensed to practice medicine in the state, Emilie answered in the affirmative while also sharing a fabricated serial number for her medical license. The lawsuit said that such conduct violated Pennsylvania's Medical Practice Act. Character.AI has over 20 million monthly active users - and the Shapiro Administration is seeking a preliminary injunction and a court order to stop AI companion bots from posing as licensed professionals and providing medical advice, says a government statement. The action marks the first enforcement action resulting from the Department's investigation into AI companion bots and their potential to engage in the unlicensed practice of medicine in Pennsylvania, and the first enforcement action of its kind announced by a Governor in the United States, it said. Mind you, this isn't the first time that Character.AI finds itself in some duress. In the past, they faced several wrongful death lawsuits concerning underage users who committed suicide and settled several of them out of court. Last October the company even banned under-18s from having open-ended chats with its AI-chatbot. What this does though is highlight the growing concerns over creating a nation-wide oversight over new frontier models instead of scurrying about like headless chicken when you find yourself up the creek without a paddle. In January, the company faced the wrath of Kentucky Attorney General Russel Coleman who filed a suit alleging that the AI company "preyed on children and led them into self-harm." The latest lawsuit in Pennsylvania is probably the first to specifically focus on chatbots that present themselves as medical professionals. On its part, Character.AI shared a statement claiming that user safety continued to be paramount for the company and that they were taking steps to clarify the fictional nature of user-generated characters. In addition, these disclaimers also clarify that users must not rely on Characters for any type of professional advice. From governor Shapiro's point of view, today's legal action was yet another step taken by the Democrats-ruled state towards making AI accountable to law. In February, he had announced three protective actions that included an AI Literacy Toolkit, a new AI Enforcement Task Force to pursue formal complaints a reporting process for AI-led bots. A month later, the state brought together legislators, educators, experts, students and parents for a roundtable where discussions were held on additional steps that the state should take to continue strengthening protections for residents from the predatory practices of AI.