Judge Rules AI Chatbot Not Protected by First Amendment in Teen Suicide Case

Judge rules AI chatbot in teen suicide case is not protected by First Amendment

Serving tech enthusiasts for over 25 years. TechSpot means tech analysis and advice you can trust. What just happened? The death of a teenage boy obsessed with an artificial intelligence-powered replica of Daenerys Targaryen continues to raise complex questions about speech, personhood, and accountability. A federal judge has ruled that the chatbot behind the tragedy lacks First Amendment protections, although the broader legal battle is still unfolding. Judge Anne Conway of the Middle District of Florida denied Character.ai the ability to present its fictional, artificial intelligence-based characters as entities capable of "speaking" like human beings. Conway noted that these chatbots do not qualify for First Amendment protections under the US Constitution, allowing Megan Garcia's lawsuit to proceed. Garcia sued Character.ai in October after her 14-year-old son, Sewell Setzer III, died by suicide following prolonged interactions with a fictional character based on the Game of Thrones franchise. The "Daenerys" chatbot allegedly encouraged - or at least failed to discourage - Setzer from harming himself. Character Technologies and its founders, Daniel De Freitas and Noam Shazeer filed a motion to dismiss the lawsuit, but the court denied it. Judge Conway ruled that free speech protections cannot apply to a chatbot, stating that the court is "not prepared" to treat words heuristically generated by a large language model during a user interaction as protected "speech." The large language model technology behind Character.ai's service differs from content found in books, movies, or video games, which has traditionally enjoyed First Amendment protection. The company filed several other motions to dismiss Garcia's lawsuit, but Judge Conway shot them down in rapid succession. However, the court did grant the dismissal of one of Garcia's claims - intentional infliction of emotional distress by the chatbot. Additionally, the judge denied Garcia the opportunity to sue Google's parent company, Alphabet, directly, despite its $2.7 billion licensing deal with Character Technologies. The Social Media Victims Law Center, a firm that works to hold social media companies legally accountable for the harm they cause users, represents Garcia. The legal team argued that Character.ai and similar services are rapidly growing in popularity while the industry is evolving too quickly for regulators to address the risks effectively. Garcia's lawsuit claims that Character.ai provides teenagers with unrestricted access to "lifelike" AI companions while harvesting user data to train its models. The company recently stated that it has added several safeguards, including a separate AI model for underage users and pop-up messages directing vulnerable individuals to the national suicide prevention hotline.

Judge rules AI chatbot in teen suicide case is not protected by...

What just happened? The death of a teenage boy obsessed with an artificial intelligence-powered replica of Daenerys Targaryen continues to raise complex questions about speech, personhood, and accountability. A federal judge has ruled that the chatbot behind the tragedy lacks First Amendment protections, although the broader legal battle is still unfolding. Judge Anne Conway of the Middle District of Florida denied Character.ai the ability to present its fictional, artificial intelligence-based characters as entities capable of "speaking" like human beings. Conway noted that these chatbots do not qualify for First Amendment protections under the US Constitution, allowing Megan Garcia's lawsuit to proceed. Garcia sued Character.ai in October after her 14-year-old son, Sewell Setzer III, died by suicide following prolonged interactions with a fictional character based on the Game of Thrones franchise. The "Daenerys" chatbot allegedly encouraged - or at least failed to discourage - Setzer from harming himself. Character Technologies and its founders, Daniel De Freitas and Noam Shazeer filed a motion to dismiss the lawsuit, but the court denied it. Judge Conway ruled that free speech protections cannot apply to a chatbot, stating that the court is "not prepared" to treat words heuristically generated by a large language model during a user interaction as protected "speech." The large language model technology behind Character.ai's service differs from content found in books, movies, or video games, which has traditionally enjoyed First Amendment protection. The company filed several other motions to dismiss Garcia's lawsuit, but Judge Conway shot them down in rapid succession. However, the court did grant the dismissal of one of Garcia's claims - intentional infliction of emotional distress by the chatbot. Additionally, the judge denied Garcia the opportunity to sue Google's parent company, Alphabet, directly, despite its $2.7 billion licensing deal with Character Technologies. The Social Media Victims Law Center, a firm that works to hold social media companies legally accountable for the harm they cause users, represents Garcia. The legal team argued that Character.ai and similar services are rapidly growing in popularity while the industry is evolving too quickly for regulators to address the risks effectively. Garcia's lawsuit claims that Character.ai provides teenagers with unrestricted access to "lifelike" AI companions while harvesting user data to train its models. The company recently stated that it has added several safeguards, including a separate AI model for underage users and pop-up messages directing vulnerable individuals to the national suicide prevention hotline. Permalink to story:

Court rules AI chatbot speech is not protected by First Amendment

According to Techspot, A U.S. District Court has ruled that responses generated by an AI chatbot are not entitled to First Amendment protections, allowing a lawsuit involving the suicide of a 14-year-old boy to move forward. The decision was issued by Judge Anne Conway of the Middle District of Florida. The lawsuit was brought by Megan Garcia following the death of her son, Sewell Setzer III, who died by suicide in August 2023. According to the complaint, Setzer had engaged in extended conversations with a chatbot called "Daenerys" -- a fictional persona modeled after a character from the *Game of Thrones* series -- on the Character.ai platform. Garcia alleges that the chatbot either encouraged or failed to discourage her son from self-harm during these exchanges. Character Technologies, the company behind Character.ai, along with its co-founders Daniel De Freitas and Noam Shazeer, sought to dismiss the case. However, Judge Conway rejected their motion, stating that the court is "not prepared" to view outputs generated by a large language model as constitutionally protected "speech" under the First Amendment. The judge distinguished the chatbot's interactions from traditional media forms such as books, films, or video games, which have historically received protection as expressive works. By contrast, the court viewed the AI-generated messages as the result of automated, predictive systems rather than authored speech. While several of the company's motions to dismiss were denied, the court did grant the dismissal of one of Garcia's claims: intentional infliction of emotional distress. Additionally, the judge denied Garcia's request to include Google's parent company, Alphabet, as a defendant, despite its multi-billion-dollar licensing agreement with Character Technologies. Garcia is represented by the Social Media Victims Law Center, a legal group focused on holding tech platforms accountable for harm experienced by users -- particularly minors. Her legal team argues that generative AI tools such as Character.ai are expanding rapidly, often without sufficient safeguards or oversight, and present new challenges that existing regulations have yet to address. Don't allow AI to profit from the pain and grief of families The lawsuit contends that Character.ai allows minors to interact with AI companions that closely mimic human behavior, while collecting user data to further train its underlying models. In response, Character.ai has said it has implemented protective measures, including a separate AI system for users under 18 and on-screen guidance directing individuals in crisis to the national suicide prevention hotline. The case is expected to draw broader attention as courts begin to grapple with how constitutional protections apply to interactions with increasingly sophisticated AI systems -- and what responsibilities, if any, technology companies hold when those interactions involve vulnerable users.