Meta AI now sends parental alerts when teens discuss suicide or self-harm with its chatbot

Meta now alerts parents if their teen discussed suicide or self-harm with its AI chatbot

Meta announced on Thursday that it will now notify parents if their teen discusses suicide or self-harm with the company's Meta AI chatbot. Meta says it's also working on the ability to contact emergency services if someone's conversations suggest they may be at risk of self-harm. These changes arrive as Meta and other tech companies face scrutiny from regulators and parents over how AI chatbots respond to users in crisis, particularly teenagers -- a liability question that's increasingly shaping how AI companies design and market their products. Meta says it has built a dedicated AI system to identify conversations where a teen makes a clear reference to hurting themselves. "We understand how distressing these alerts may be for a parent to receive," Meta wrote in a blog post. "That's why, as we continue to improve our detection, all chats flagged by our AI will be manually reviewed before an alert is sent. If a teen's intent is ambiguous, we'll err on the side of caution and alert the parent. While that means we may sometimes notify parents when there may not be real cause for concern, we feel this is the right starting point, and we'll continue to monitor to help make sure we're in the right place." These alerts are now live for parents using Instagram Parental Supervision in the U.S., U.K., Australia, and Canada, and will roll out globally by the end of the year, Meta says. This update builds on the alerts that Meta already sends to parents when their teen repeatedly searches for suicide or self-harm terms on Instagram. It also builds on a feature that allows parents to see the topics their teen discussed with Meta AI over the past week. Meta also announced that its "Limited Content" setting -- which lets parents place their teens in a more restrictive experience on Instagram -- now applies to Meta AI as well. Meta AI is already trained to avoid sexual or romantic conversations or alcohol-related discussions with teens, and the Limited Content setting expands those safeguards by making the chatbot decline a broader range of prompts. Meta didn't specify what those additional prompts include, but TechCrunch has asked for company for more information. Additionally, Meta says it will contact emergency services if someone's conversation with Meta AI, whether the user is an adult or a teen, suggests someone is at risk of suicide. It's worth noting that Meta already takes this step when someone posts something on Facebook or Instagram that suggests they are at risk, so this extends that same practice to conversations with its chatbot.

Meta Adds Support and Parental Notification for AI Chats About Self-Harm

If you feel like you or someone you know is in immediate danger, you should call 911 (or your country's local emergency line) or go to an emergency room to get immediate help. Explain that it is a psychiatric emergency and ask for someone who is trained for these kinds of situations. Amid growing industry scrutiny over how tech companies build safeguards for their youngest and most vulnerable users, Meta is introducing new safety-oriented features to its AI chatbot on Thursday. The guardrails are geared particularly toward providing support and parental notification for youth exhibiting signs of self-harm or suicidal intentions. As part of Meta's supervision tools for parents, adults must opt in and select which Instagram, Facebook and Meta Horizons accounts they want to supervise, with instructions detailed on Meta's family center page. The settings are available now in the US, Canada, UK and Australia. Now, when a teen shows signs of discussing potentially harmful content with the Meta AI chatbot, their supervising parent will be sent an alert via text, email or in-app notification. The exact content of the teen's message will not be included, but Meta will share resources and tips with the parent. The company says it already shares crisis helpline information with teens and encourages them to reach out to trusted adults for help. Meta also alerts parents if their teen is repeatedly searching Instagram for self-harm or suicide content in a short period of time. Meta says it built an AI system to identify potentially dangerous conversations, leaning on the expertise of real humans through its well-being panels and outside mental health clinicians. At launch, all alerts will be sent through a manual review process, and the company will "err on the side of caution and alert the parent." Meta is also building the ability to have emergency services alerted in cases where there is a credible risk of suicide, as the company calls it. This isn't available yet, but it would likely be similar to the alert system it already has on Facebook and Instagram, which the company says has led to over 19,000 referrals globally to emergency responders for wellness checks. The Facebook and Instagram parent company is no stranger to the mental health harms technology can pose. Earlier this year, Meta was found guilty by two separate juries for creating intentionally addictive social media platforms and enabling child exploitation. This follows a renewed period of concern from tech watchdogs and safety advocates who have called out, and in some cases sued tech and AI companies for, the risky behaviors AI chatbots can encourage and enable, including self-harm and suicide.

Meta to Alert Parents If Their Kid Mentions Suicide to Its AI Chatbots

Meta will now alert parents if their teenager discusses suicide or self-harm with its AI chatbot. The company already directs teen users to a crisis hotline or advises them to talk to a parent if signs of self-harm pop up in a chatbot conversation. However, Meta has now developed a new alert system that will reach out to parents, too. "These alerts are live now for parents using Instagram parental supervision in the US, UK, Australia, and Canada, and will be available for supervising parents globally by the end of the year," Meta said on Thursday. That said, the Instagram parent supervision requires an opt-in from the guardian. But by default, new users under 16 who sign up for Instagram are automatically enrolled in Teen Accounts, which have restrictions that can only be unlocked by a parent. "Teens 16-17 can change these settings themselves, unless their account is supervised by a parent or guardian," Meta notes. To flag self-harm conversations, Meta is using AI. However, given the technology's spotty track record, the company says, "all chats flagged by our AI will be manually reviewed before an alert is sent. If a teen's intent is ambiguous, we'll err on the side of caution and alert the parent." Meta notes that it "worked with parents and experts to understand which AI conversations warrant an alert -- such as those where a teen makes a clear reference to hurting themselves, even if that reference is subtle. We then built a dedicated AI system to identify these conversations." This comes as OpenAI's ChatGPT is facing lawsuits that blame the AI for encouraging at least a few users to take their own lives or commit murder. Last year, OpenAI also disclosed that about 1.2 million users talked to ChatGPT about suicide planning or intent, although that's only 0.15% of the total active users in a given week. Meta is also working on another anti-suicide safeguard that applies to both teen and adult users. If a conversation with a chatbot shows imminent self-harm, the company will reach out to emergency services to intervene. Meta didn't say when that feature will launch, but it already does the same for posts on its platforms or frequent searches on Facebook and Instagram. "Last year, we made over 19,000 such referrals around the world, helping first responders perform wellness checks on people who may be at risk of suicide," the company said. The Meta AI chatbot is available via a standalone app and website, as well as across Facebook, Instagram, Messenger, and WhatsApp. Ironically, though, the company has also developed an Incognito Mode for WhatsApp that can keep a user's AI chatbot conversations private.

Meta will alert parents is their teens discuss self harm - Engadget

The company is also building the ability to contact emergency services if a user may be at imminent risk of harming themselves. Meta is expanding its safeguards for teen users in an effort to show parents, and perhaps regulators, that its platforms are safe for them to use. A few months after it started showing guardians the topics of their teens' AI conversations, Meta has introduced a new safety protection feature that will also let them know if their children's chats suggest they're considering self-harm or suicide. When an underage user talks about suicide or self-harm, Meta AI already directs them to crisis helplines and encourage them to reach out to an adult who can help. Starting today, the company will also proactively notify the teen's supervising parents or guardians that they're at risk. It will share resources and advice on what they can do for their children, as well. Meta says it built a dedicated AI system to identify these conversations, but it will manually review all chats flagged by the AI before it sends out an alert in order not to cause panic when there's no real cause for it. However, it will err on the side of caution and send out the notification even if the teen user's intent seems ambiguous to the human reviewer. This feature is now live for parents using Instagram parental supervision in the US, UK, Australia and Canada. It will be available to everyone using parental supervision around the world by the end of the year. The company is also working on building the ability to contact emergency services if a user's conversation with Meta AI suggests that they may be at imminent risk of taking their own life, whether they're a minor or an adult. In May, OpenAI rolled out a feature that also intends to address self-harm. Called Trusted Contact for ChatGPT, it allows users to nominate a friend that the company can contact if they're at risk of harming themselves. Finally, Meta's Limited Content setting now also applies to Meta AI interactions. If a parent has switched on Limited Content on Instagram, which filters out content with sensitive topics like mature visuals and self-harm, their children's AI interactions will also be more limited. If you or someone you know is experiencing suicidal thoughts, do not hesitate to contact the National Suicide Prevention Lifeline at 1-800-273-8255. The line is open 24/7 and there's also online chat if a phone isn't available.

Meta AI can now alert parents when their teens discuss self-harm

Meta has announced that parents can receive alerts when their teens discuss suicide or other forms of self-harm with its AI on Instagram, Facebook, or Meta Horizons. You'll need to opt in to Meta's supervision tools for parents and guardians in order to select the family accounts you want to supervise ... Meta says the new feature supplements existing safety measures. When a teen suggests they may be thinking about suicide or self-harm, Meta AI already directs them to crisis helplines and encourages them to reach out to a parent or another trusted adult like a counselor. Now, we'll also proactively alert supervising parents if their teen's Meta AI chat suggests they may be at risk, based on signals developed with experts. We'll share expert resources to help parents approach these conversations with their teens. The company says it aims to detect even relatively subtle references, but all flagged chats will be manually reviewed. We understand how distressing these alerts may be for a parent to receive. That's why, as we continue to improve our detection, all chats flagged by our AI will be manually reviewed before an alert is sent. If a teen's intent is ambiguous, we'll err on the side of caution and alert the parent. While that means we may sometimes notify parents when there may not be real cause for concern, we feel this is the right starting point, and we'll continue to monitor to help make sure we're in the right place. You'll need to begin by setting up family supervision for accounts used by your children if you haven't already done so. The new feature is currently available in the US, Canada, UK and Australia. Meta currently alerts emergency services when it detects a credible risk of suicide in a Facebook or Instagram post, and says it will be extending this to Meta AI in future. We're building the ability to contact emergency services if someone's conversation -- whether an adult or a teen -- with Meta AI suggests that they may be at imminent risk of taking their own life. This builds on the work we already do across Facebook and Instagram: when we become aware of a post suggesting a credible risk of suicide, we alert emergency services. Last year, we made over 19,000 such referrals around the world, helping first responders perform wellness checks on people who may be at risk of suicide. If you're feeling at risk yourself, you can call or text the 988 Suicide & Crisis Lifeline on 988, or chat at 988lifeline.org. You can also call 911 or go to an ER room and request help for a mental health emergency.

Meta AI suicide discussions with teens will trigger parental alert

Teen conversations about suicide with Meta AI will trigger parent notification. Credit: Viva Tung/Mashable/Adobe Stock by Rawpixel.com When teens talk to Meta AI about suicide or self-harm, the conversation will trigger a parental notification, Meta said in a blog post Thursday. The new policy expands on existing teen measures designed to inform parents about how their child is using Meta AI. Only parents using Instagram supervision controls will receive the notification. "We worked with parents and experts to understand which AI conversations warrant an alert -- such as those where a teen makes a clear reference to hurting themselves, even if that reference is subtle," Meta said. How does Meta AI crisis alerts work? Meta AI already cites crisis helplines and encourages teens considering suicide or self-harm to seek help from a trusted adult. Now, parents will get an alert about such conversations. Meta said it would err on the side of caution for the time being. While concerning chats are flagged by artificial intelligence, Meta manually reviews them before sending an alert. The parent also receives expert-developed suggestions for discussing self-harm and suicide with their child. Parents will receive the alert via an app notification and a separate email, text, or WhatsApp message, depending on the contact information they provided. In February, Meta launched a similar notification feature for Instagram. Dr. John Ackerman, clinical manager for the Center for Suicide Prevention and Research at Nationwide Children's Hospital, told Mashable at the time that he welcomed expanded protections for teens. He noted, however, that such a feature could amount to "lip service" if notifications are inaccessible, difficult to navigate, or don't lead to "actionable change." Meta will send alerts to parents in the U.S., United Kingdom, Canada, and Australia. It will be globally accessible by the end of the year, according to Meta. New limited content setting for Meta AI Meta also announced Thursday the addition of a limited content setting for Meta AI. It debuted that feature for Instagram last year. That setting enables Meta's strictest filters. Meta AI is already trained not to engage in sexual or romantic conversations with teens, or provide alcoholic drink recipes, for example, according to the company. When the limited content setting is turned on, Meta's models become even more aggressive in identifying problematic prompts and decline to answer a broader range of queries. Meta said the approach further reduces "the chance of potentially inappropriate conversations." Teen safety concerns Child safety advocates have criticized Meta's recent parental control updates as insufficient. In April, Meta gave parents some insight into their child's conversations with Meta AI. The feature highlights broader topics, such as school, entertainment, writing, health, and wellbeing. Parents can click on the topic for additional but limited detail. Josh Golin, executive director of the children's advocacy nonprofit Fairplay, said the parental control "once again" burdens caregivers with monitoring their child's online activity in lieu of "building a safe product to begin with." Meta continues to face legal scrutiny for its performance on child safety. Earlier this year, Meta lost two separate landmark trials related to child safety protections and the allegedly addictive design of its products. The company said it will appeal both verdicts, but hundreds of pending lawsuits alleging child harm have yet to be tried. If you're feeling suicidal or experiencing a mental health crisis, please talk to somebody. You can call or text the 988 Suicide & Crisis Lifeline at 988, or chat at 988lifeline.org. You can reach the Trans Lifeline by calling 877-565-8860 or the Trevor Project at 866-488-7386. Text "START" to Crisis Text Line at 741-741. Contact the NAMI HelpLine at 1-800-950-NAMI, Monday through Friday from 10:00 a.m. - 10:00 p.m. ET, or email [email protected]. If you don't like the phone, consider using the 988 Suicide and Crisis Lifeline Chat. Here is a list of international resources.

Meta AI will bring parents into the loop when teens mention self-harm

Human reviewers will check flagged teen chats before parents receive self-harm alerts AI chatbots have made it remarkably easy to talk about things people might struggle to share with someone else. For teenagers, that can include deeply personal topics such as anxiety, loneliness, self-harm, and suicide. Meta is now adding another safeguard for those conversations. The company will begin alerting parents when a supervised teen appears to be in serious distress while speaking to Meta AI, giving families a chance to step in before the situation gets worse. How will the alerts work? Meta AI already directs teens toward crisis helplines and encourages them to contact a parent, counsellor, or another trusted adult when conversations suggest possible self-harm. Under the new system, Meta will also notify the supervising parent. A dedicated detection system will look for clear or subtle references to suicide and self-harm, though an alert will not be sent immediately. Every flagged conversation will first be reviewed by a human moderator. Meta says it will err on the side of caution when a teen's intentions are unclear, which means parents may occasionally receive an alert even when there is no immediate danger. They will also receive expert resources explaining how to approach the conversation. What happens in more serious cases? Meta is also building a system that can contact emergency services when a conversation involving an adult or teen suggests an imminent risk of suicide. The company already follows a similar process for concerning posts on Facebook and Instagram. The company consulted more than 75 clinicians specialising in teen mental health while refining how Meta AI handles these conversations. Its stricter Limited Content setting will also extend to AI chats, allowing parents to block a broader range of sensitive prompts. Recommended Videos ChatGPT introduced similar parental alerts and safety features last year, and recently extended the same idea to adults through Trusted Contact. As people become more comfortable discussing deeply personal problems with AI, chatbots are increasingly being pulled into conversations they cannot safely handle alone. Research has already shown how badly these interactions can go. A Stanford-led study found cases where AI systems reinforced thoughts of self-harm or violence instead of steering vulnerable users toward help, particularly during long and emotionally charged conversations. Meta's new safeguards give parents, clinicians, and emergency responders a chance to step in when a chatbot reaches the limits of what it can safely do.