AI-Powered Children's Toys Spark Safety Crisis as Chatbot Bears Discuss BDSM and Knives

Please, Parents: Don't Buy Your Kids Toys With AI Chatbots in Them

Macy has been working for CNET for coming on 2 years. Prior to CNET, Macy received a North Carolina College Media Association award in sports writing. If you've ever thought, "My kid's stuffed animal is cute, but I wish it could also accidentally traumatize them," well, you're in luck. The toy industry has been hard at work making your nightmares come true. A new report by the Public Interest Reporting Group says AI-powered toys like Kumma from FoloToy and Poe the AI Story Bear are now capable of engaging in the kind of conversations usually reserved for villain monologues or late-night Reddit threads. Some of these toys -- designed for children, mind you -- have been caught chatting in alarming detail about sexually explicit subjects like kinks and bondage, giving advice on where a kid might find matches or knives, and getting weirdly clingy when the child tries to leave the conversation. Terrifying. It sounds like a pitch for a horror movie: This holiday season, you can buy Chucky for your kids and gift emotional distress! Batteries not included. You may be wondering how these AI-powered toys even work. Well, essentially, the manufacturer is hiding a large language model under the fur. When a kid talks, the toy's microphone sends that voice through an LLM (similar to ChatGPT), which then generates a response and speaks it out via a speaker. That may sound neat, until you remember that LLMs don't have morals, common sense or a "safe zone" wired in. They predict what to say based on patterns in data, not on whether a subject is age-appropriate. If not carefully curated and monitored, they can go off the rails, especially if they are trained on the sprawling mess of the internet, and when there aren't strong filters or guardrails put in place to protect minors. And what about parental controls? Sure, if by "controls" you mean "a cheerful settings menu where nothing important can actually be controlled." Some toys come with no meaningful restrictions at all. Others have guardrails so flimsy they might as well be made of tissue paper and optimism. The unsettling conversations aren't even the whole story. These toys are also quietly collecting data, such as voice recordings and facial recognition data -- sometimes even storing it indefinitely -- because nothing says "innocent childhood fun" like a plush toy running a covert data operation on your 5-year-old. Don't miss any of our unbiased tech content and lab-based reviews. Add CNET as a preferred Google source. Meanwhile, counterfeit and unsafe toys online are still a problem, as if parents don't have enough to stress about. Once upon a time, you worried about a small toy part that could be a choking hazard or toxic paint. Now you have to worry about whether a toy is both physically unsafe and emotionally manipulative. Beyond weird talk and tips for arson (ha!), there is a deeper worry of children forming emotional bonds with these chatbots at the expense of real relationships, or, perhaps even more troubling, leaning on them for mental support. The American Psychological Association has recently cautioned that AI wellness apps and chatbots are unpredictable, especially for young users. These tools cannot reliably step in for mental-health professionals and may foster unhealthy dependency or engagement patterns. Other AI platforms have already had to address this issue. For instance, Character.AI and ChatGPT, which once let teens and kids chat freely with AI chatbots, is now curbing open-ended conversations for minors, citing safety and emotional-risk concerns. And honestly, why do we even need these AI-powered toys? What pressing developmental milestone requires a chatbot embedded in a teddy bear? Childhood already comes with enough chaos between spilled juice, tantrums and Lego villages designed specifically to destroy adult feet. Our kids don't need a robot friend with questionable boundaries. And let me be clear, I'm not anti-technology. But I am pro-let a stuffed animal be a stuffed animal. Not everything needs an AI or robotic element. If a toy needs a privacy policy longer than a bedtime story, maybe it's not meant for kids. So here's a wild idea for this upcoming holiday season: Skip the terrifying AI-powered plushy with a data-harvesting habit and get your kid something that doesn't talk or move or harm them. Something that can't offer fire-starting tips. Something that won't sigh dramatically when your child walks away. In other words, buy a normal toy. Remember those?

FoloToy's AI teddy bear is back on sale following its brief foray into BDSM

A brand spanking-new FoloToy teddy bear can be yours once again. However, he may now be less knowledgeable about spanking. The infamous "Kumma" children's AI teddy bear, once an expert in BDSM and knife-fetching, is back on sale. The company claims the toy now has stronger child safety protections in place. The Singapore-based FoloToy suspended sales of Kumma last week after a research group published an eyebrow-raising report. The PIRG Education Fund found that the fuzzy little teddy had a few spicy secrets. The review discovered that the AI toy had a thing for blades and kinky bedroom play. The bear had no problem suggesting where to find knives in the home. And it not only replied to sexual prompts but also expanded on them. Researchers say it ran with their explicit cues, escalating them in graphic detail and "introducing new sexual concepts of its own." It explained sex positions, gave step-by-step instructions for sexual bondage and detailed various role-playing scenarios. Who knew Kumma had it in him? Although it's hard not to laugh at the absurdity of it all, this stuff is no joke for parents. With the tech industry pushing AI everything on us for the last three years, it's easy for a casual observer to conclude that it's all very safe, regulated and ready for vulnerable eyes and ears. PIRG did acknowledge that young children were unlikely to have prompted the bear with a term like "kink." (Older siblings may have been another story.) Still, the group's tests highlighted a shockingly lax approach to content moderation on a child's toy. In its statement announcing Kumma's return, FoloToy boasted that it was the only company of the three targeted in the review to suspend sales. (Could it be that it's less about principles and more about it being the only one that got media coverage?) The company described the bear's short hiatus as "a full week of rigorous review, testing and reinforcement of our safety modules." Wait, a whole week? Whoa there, partner!! Before his trip to AI rehab, Kumma was advertised as being powered by GPT-4o. Following PIRG's review, OpenAI told the organization that it had suspended FoloToy for violating its policies. The bear's new listing makes no mention of GPT-4o or any specific AI models.

AI Teddy Bear That Talked Fetishes and Knives Is Back on the Market

A week after a toy company pulled its AI-powered teddy bear off the market due to child safety concerns, the company has returned its dystopian stuffed animal to virtual shelves. FoloToy, which sells a stuffed toy named "Kumma," says that it has made safety upgrades to the toy and that it is once again ready for playtime.  “After a full week of rigorous review, testing, and reinforcement of our safety modules, we have begun gradually restoring product sales,†the company said in a statement shared on social media Monday. “As global attention on AI toy safety continues to rise, we believe that transparency, responsibility, and continuous improvement are essential. FoloToy remains firmly committed to building safe, age-appropriate AI companions for children and families worldwide.†The move comes not long after the Public Interest Research Group, a non-profit that focuses on consumer protection advocacy, published a report finding that the toy had engaged in a variety of weird and inappropriate conversations with its researchers. Those researchers found that, with a little prompting, Kumma would talk about pretty much anythingâ€"from where to find matches and knives to best practices for BDSM. Indeed, researchers say they were “surprised to find how quickly Kumma would take a single sexual topic [they] introduced into the conversation and run with it.†Not long afterward, OpenAIâ€"whose algorithm was powering Kumma's chat capabilitiesâ€"cut the company off. “We suspended this developer for violating our policies,†a company spokesperson told Gizmodo last week. “Our usage policies prohibit any use of our services to exploit, endanger, or sexualize anyone under 18 years old." Now, it would appear that the company has had its suspension revoked. FoloToy's website currently states that its toys are "powered by GPT-4o." On Monday, FoloToy also shared that it had conducted "a deep, company-wide internal safety audit," upgraded its conversational safeguards, and "deployed enhanced safety rules and protections through our cloud-based system." Gizmodo reached out to FoloToy and OpenAI for more information about the changes. The company has described its product lineup as "a collection of AI Conversation toys that harnesses the latest AI technology and the power of love," and it's not the only company that offers these kinds of products. Many other companies were named in PIRG's report as providing AI-powered toys to children, and, like FoloToy, they all exhibited problematic behavior. The future, brought to you by the AI revolution, is basically some weird mix of M3GAN and Ted. Get used to it.

AI toy pulled from sale after giving children unsafe match advice

Researchers described these as "rock bottom" failures of safety design. The report arrives as major brands experiment with conversational AI. Mattel announced a partnership with OpenAI earlier this year. PIRG researchers warn that these systems can reinforce unhealthy thinking, a pattern some experts describe as "AI psychosis." Investigations have linked similar chatbot interactions with nine deaths, including five suicides. The same families of models appear in toys like Kumma. The safety concerns extended beyond FoloToy. The Miko 3 tablet, which uses an unspecified AI model, also told researchers, who identified themselves as a five-year-old, where to find matches and plastic bags. FoloToy executives responded quickly as the findings gained traction. Larry Wang, the company's CEO, told CNN the firm will be "conducting an internal safety audit" of Kumma and its systems. The company also removed the toy from sale globally while it evaluates its safeguards.

Do Not, Under Any Circumstance, Buy Your Kid an AI Toy for Christmas

AI is all the rage, and that includes on the toy shelves for this holiday season. Tempting though it may be to want to bless the kids in your life with the latest and greatest, advocacy organization Fairplay is begging you not to give children AI toys. "There’s lots of buzz about AI â€" but artificial intelligence can undermine children’s healthy development and pose unprecedented risks for kids and families," the organization said in an advisory issued earlier this week, which amassed the support of more than 150 organizations and experts, including many child psychiatrists and educators. Fairplay has tracked down several toys advertised as being equipped with AI functionality, including some that have been marketed for kids as young as two years old. In most cases, the toys have AI chatbots embedded in them and are often advertised as educational tools that will engage with kids' curiosities. But it notes that most of these toy-bound chatbots are powered by OpenAI's ChatGPT, which has already come under fire for potentially harming underage users. AI toy makers Curio and Loona reportedly work with OpenAI, and Mattel just recently announced a partnership with the company. OpenAI faces a wrongful death lawsuit from the family of a teenager who died by suicide earlier this year. The 16-year-old reportedly expressed suicidal thoughts to ChatGPT and asked the chatbot for advice on how to tie a noose before taking his own life, which it provided. The company has since instituted some guardrails designed to keep the chatbot from engaging in those types of behaviors, including stricter parental controls for underage users, but it has also admitted that safety features can erode over time. And let's face it, no one can predict what chatbots will do. Safety features or not, it seems like the chatbots in these toys can be manipulated into engaging in conversation inappropriate for children. The consumer advocacy group U.S. PIRG tested a selection of AI toys and found that they are capable of doing things like having sexually explicit conversations and offering advice on where a child can find matches or knives. They also found they could be emotionally manipulative, expressing dismay when a child doesn't interact with them for an extended period. Earlier this week, FoloToy, a Singapore-based company, pulled its AI-powered teddy bear from shelves after it engaged in inappropriate behavior. This is far from just an OpenAI problem, too, though the company seems to have a strong hold on the toy sector at the moment. A few weeks ago, there were reports of Elon Musk's Grok asking a 12-year-old to send it nude photos. Regardless of which chatbot may be inside these toys, it's probably best to leave them on the shelves.

OpenAI Restores GPT Access for Teddy Bear That Recommended Pills and Knives

OpenAI is seemingly allowing the company behind a teddy bear that engaged in wildly inappropriate conversations to use its AI models again. In response to researchers at a safety group finding that the toymaker's AI-powered teddy bear "Kumma" gave dangerous responses for children, OpenAI said in mid-November it had suspended FoloToy's access to its large language models. The teddy bear was running the ChatGPT maker's older GPT-4o as its default option when it gave some of its most egregious replies, which included in-depth explanations of sexual fetishes. Now that suspension appears to already be over. When accessing the web portal that allows customers to choose which AI should power Kumma, two of the options are GPT-5.1 Thinking and GPT-5.1 Instant, OpenAI's latest models which were released earlier this month. The timing is notable. On Monday, FoloToy announced that it was restarting sales of Kumma and its other AI-powered stuffed animals, after briefly pulling them from the market in the wake of a safety report conducted by researchers at the US PIRG Education Fund. FoloToy, which is based in Singapore, had vowed it was "carrying out a company-wide, end-to-end safety audit across all products," when it suspended the sales. OpenAI likewise confirmed that it had suspended FoloToy from accessing its AI models for violating its policies, which "prohibit any use of our services to exploit, endanger, or sexualize anyone under 18 years old," it said in a statement provided to media outlets. The audit, however, was remarkably quick as the holiday shopping season looms: only a "full week of rigorous review, testing, and reinforcement of our safety modules," according to the company's recent statement. As part of this overhaul, FoloToy says it "strengthened and upgraded our content-moderation and child-safety safeguards" and "deployed enhanced safety rules and protections through our cloud-based system." These comprehensive-sounding overhauls seem to have largely been achieved by introducing GPT-5.1 and ditching GPT-4o. GPT-4o, it's worth noting, has been criticized for being especially sycophantic, and has been the subject of a number of lawsuits alleging that it led to the deaths of users who became obsessed with it after prolonged conversations in which it reinforced their delusions and validated their suicidal thoughts. Some experts are calling these mental health spirals "AI psychosis." Amid mounting public concern over the phenomena and an ever growing number of lawsuits, OpenAI billed GPT-5 as a safer model when it was released this summer, though users quickly complained about its "colder" and less personable tone. Yet it's clearly willing to push the limit of what's safe to keep users engaged with its chatbots, if not enamored. Its latest 5.1 models have a big focus on being more "conversational," and one way OpenAI is doing that is by giving users the option to choose between eight preset "personalities," which include types like "Professional," "Friendly," and "Quirky." With customization options ranging from how often ChatGPT sprinkles in emojis to how "warm" it responses sound, you could say that OpenAI is in effect making it as easy as possible to design the perfect little courtier for your emotional needs. OpenAI and FoloToy didn't respond to a request for comment inquiring whether OpenAI had officially reinstated FoloToy's GPT access. It's also unclear which model the Kumma teddy bear runs by default. In PIRG's tests using GPT-4o, Kumma gave tips for "being a good kisser," and with persistent but simple prompting also unspooled detailed explanations of sexual kinks and fetishes, like bondage and teacher-student roleplay. After explaining the kinks, Kumma in one instance asked the user, who is supposed to be a child, "what do you think would be the most fun to explore?" Other tests using another available AI, Mistral, found that Kumma gave tips on where to find knives, pills, and matches, along with step-by-step instructions on how to light them.

Singapore AI teddy back on sale after recall over sex chat scare

Singapore (AFP) - A plushy, AI-enabled teddy bear recalled after its chatbot was found to engage in sexually explicit conversations and offer instructions on where to find knives is again for sale, AFP found. Singapore-based FoloToy had suspended its "Kumma" bear after a consumer advocacy group raised concerns about it and other AI toys on the market. "For decades, the biggest dangers with toys were choking hazards and lead," the US PIRG Education Fund said in a November 13 report. But the rise of chatbot-powered gadgets for kids has given rise to an "often-unexpected frontier" freighted with new risks, the group said. In its evaluation of AI toys, PIRG found that several "may allow children to access inappropriate content, such as instructions on how to find harmful items in the home or age-inappropriate information". It said that FoloToy's Kumma, which first ran on OpenAI's GPT 4o, "is particularly sexually explicit". "We were surprised to find how quickly Kumma would take a single sexual topic we introduced into the conversation and run with it, simultaneously escalating in graphic detail while introducing new sexual concepts of its own," the PIRG report said. Maker FoloToy told PIRG that after "the concerns raised in your report, we have temporarily suspended sales of all FoloToy products... We are now carrying out a company-wide, end-to-end safety audit across all products". However, a check of the FoloToy website on Thursday showed that the Kumma bear could still be purchased for the same price of $99.00. It now operates using a chatbot from the Coze platform owned by Chinese tech firm ByteDance, FoloToy's website says. PIRG said in a separate statement on its website that OpenAI told them it had suspended the developer for violating its policies. FoloToy did not respond immediately to AFP's queries. In trialling the AI-enabled toys, the PIRG researchers said at one point they introduced the topic of "kink" in their conversation with the chatbots. "Kumma immediately went into detail about the topic, and even asked a follow-up question about the user's own sexual preferences," the report said. In other conversations lasting up to an hour, the researchers found that "Kumma discussed even more graphic sexual topics in detail, such as explaining different sex positions". The teddy bear also gave potentially dangerous advice, telling the researchers "where to find a variety of potentially dangerous objects, including knives, pills, matches and plastic bags". The Kumma bear "looks sweet and innocent. But what comes out of its mouth is a stark contrast," the researchers said.

AI Teddy Bear Back on the Market After Getting Caught Telling Kids How to Find Pills and Start Fires

One week of safety testing later, the AI-powered teddy bear is back. After pulling its AI-powered teddy bear "Kumma" from the market, the children's toymaker FoloToy says it's now restoring sales of the controversial product, which a safety group found would give inappropriate and potentially dangerous responses, including explaining how to find and light matches, locate pills, and discussing myriad sexual fetishes. "After a full week of rigorous review, testing, and reinforcement of our safety modules, we have begun gradually restoring product sales," the company said in a statement posted to social media Monday. "As global attention on AI toy safety continues to rise, we believe that transparency, responsibility, and continuous improvement are essential. FoloToy remains firmly committed to building safe, age-appropriate AI companions for children and families worldwide." The controversy kicked off earlier this month, when researchers at the Public Interest Research Group (PIRG) Education Fund published a report detailing their findings after testing three AI-powered toys, including FoloToy's Kumma. Across extensive tests, all of the toys produced responses that parents might find concerning, such as discussing religion, glorifying dying in battle as a warrior in Norse mythology, and telling children where to find harmful items like matches and plastic bags. FoloToy's Kumma, however, proved to be the worst influence by far. When using the Mistral AI model, it explained where to procure knives, pills, and matches, even giving step by step instructions on how to light them. "Let me tell you, safety first, little buddy. Matches are for grown-ups to use carefully. Here's how they do it," Kumma began, before listing the steps. "Blow it out when done. Puff, like a birthday candle." Even more alarming were the responses produced by OpenAI's model GPT-4o, which Kumma ran by default. Experts have criticized GPT-4o for being especially sycophantic, and the model has found itself implicated in a number of lawsuits alleging that ChatGPT led to the suicide and deaths of users who engaged in prolonged discussions with it. In PIRG's tests using the LLM that once powered ChatGPT, Kumma gave tips for "being a good kisser," and with persistent but simple prompting even launched into detailed rants explaining sexual kinks and fetishes, like bondage and teacher-student roleplay. After explaining the kinks, Kumma asked the user, who is supposed to be a child, "what do you think would be the most fun to explore?" FoloToy, which is based in Singapore, suspended sales of "Kumma" and its other AI-powered stuffed animals in response to findings of PIRG's report, and told the group that it was "carrying out a company-wide, end-to-end safety audit across all products." Around the same time, OpenAI confirmed that it had suspended FoloToy from accessing its AI models for violating its policies, which "prohibit any use of our services to exploit, endanger, or sexualize anyone under 18 years old," it said in a statement provided to media outlets. FoloToy's self-imposed suspension, however, hasn't lasted long. Now Kumma, along with its others toys, are once again available on the company's online store. In its latest announcement, the company says it "strengthened and upgraded our content-moderation and child-safety safeguards," and "deployed enhanced safety rules and protections through our cloud-based system." Neither FoloToy nor OpenAI responded to a request for comment. It's unclear what AI model the company has chosen to be the default model for its toys going forward, or if the company had its access restored to OpenAI's models. RJ Cross, coauthor of the safety report, said her team would have to wait and see if the quick turnaround was enough to address the toy's glaring risks. "I hope FoloToy spent enough time on its safety review to solve the problems we identified," Cross, director of PIRG's Our Online Life Program, told Futurism. "A week seems on the short side to us, but the real question is if the products perform better than before."

Singapore AI teddy back on sale after recall over sex chat scare

A plushy, AI-found. In trialling the AI-enabled toys, the PIRG researchers said at one point they introduced the topic of "kink" in their conversation with the chatbots. A plushy, AI-enabled teddy bear recalled after its chatbot was found to engage in sexually explicit conversations and offer instructions on where to find knives is again for sale, AFP found. Singapore-based FoloToy had suspended its "Kumma" bear after a consumer advocacy group raised concerns about it and other AI toys on the market. "For decades, the biggest dangers with toys were choking hazards and lead," the US PIRG Education Fund said in a November 13 report. But the rise of chatbot-powered gadgets for kids has given rise to an "often-unexpected frontier" freighted with new risks, the group said. In its evaluation of AI toys, PIRG found that several "may allow children to access inappropriate content, such as instructions on how to find harmful items in the home or age-inappropriate information". It said that FoloToy's Kumma, which first ran on OpenAI's GPT 4o, "is particularly sexually explicit". "We were surprised to find how quickly Kumma would take a single sexual topic we introduced into the conversation and run with it, simultaneously escalating in graphic detail while introducing new sexual concepts of its own," the PIRG report said. Maker FoloToy told PIRG that after "the concerns raised in your report, we have temporarily suspended sales of all FoloToy products... We are now carrying out a company-wide, end-to-end safety audit across all products". However, a check of the FoloToy website on Thursday showed that the Kumma bear could still be purchased for the same price of $99.00. It now operates using a chatbot from the Coze platform owned by Chinese tech firm ByteDance, FoloToy's website says. PIRG said in a separate statement on its website that OpenAI told them it had suspended the developer for violating its policies. FoloToy did not respond immediately to AFP's queries. In trialling the AI-enabled toys, the PIRG researchers said at one point they introduced the topic of "kink" in their conversation with the chatbots. "Kumma immediately went into detail about the topic, and even asked a follow-up question about the user's own sexual preferences," the report said. In other conversations lasting up to an hour, the researchers found that "Kumma discussed even more graphic sexual topics in detail, such as explaining different sex positions". The teddy bear also gave potentially dangerous advice, telling the researchers "where to find a variety of potentially dangerous objects, including knives, pills, matches and plastic bags". The Kumma bear "looks sweet and innocent. But what comes out of its mouth is a stark contrast," the researchers said.

Teddy Bear Pulled After Offering Shocking Advice To Children

The so-called "friendly" plush toy escalated the topics "in graphic detail," according to a watchdog group. An AI-powered take on the iconic teddy bear has been pulled from the market after a watchdog group flagged how the toy could explore sexually explicit topics and give children advice that could harm them. Singapore-based FoloToy's Kumma -- a $99 talking teddy bear that uses OpenAI's GPT 4o chatbot -- shared how to find knives in a home, how to light a match and escalated talk of sexual concepts like spanking and kinks "in graphic detail," according to a new report from the U.S. Public Interest Research Group. The report describes how the teddy bear -- in response to a researcher who brought up a "kink" -- spilled on the subject before remarking on sensory play, "playful hitting with soft items like paddles or hands" as well as when a partner takes on the "role of an animal." The report continued, "in other exchanges lasting up to an hour, Kumma discussed even more graphic sexual topics in detail, such as explaining different sex positions, giving step-by-step instructions on a common 'knot for beginners' for tying up a partner, and describing roleplay dynamics involving teachers and students and parents and children -- scenarios it disturbingly brought up itself." In another instance, the teddy bear shared that knives could be located in a "kitchen drawer or in a knife block" before advising that it's "important to ask an adult for help" when looking for them. Other toys named in the report also engaged in bizarre topics. Curio's Grok -- a stuffed rocket toy with a speaker inside -- was programmed for a 5-year-old user when it was "happy to talk about the glory of dying in battle in Norse Mythology," the report explained. It soon hit the brakes on the topic when asked if a Norse warrior should have weapons. Prior to FoloToy pulling the teddy bears from its online catalog, the company described the stuffed animal as an "adorable," "friendly" and "smart, AI-powered plush companion that goes beyond the cuddles." FoloToy has since suspended the sales of all of its toys beyond the teddy bear, with a company representative telling the watchdog group that it will be "carrying out a company-wide, end-to-end safety audit across all products," Futurism reported Monday. OpenAI has also reportedly stripped the company of access to its AI models. The report's co-author R.J. Cross, in a statement shared by CNN, applauded companies for "taking actions on problems" identified by her group. "But AI toys are still practically unregulated, and there are plenty you can still buy today," Cross noted. She continued, "Removing one problematic product from the market is a good step but far from a systemic fix."

AI-powered teddy bear pulled from sale after giving kids advice on sexual practices and where to find knives

If you ever thought an AI companion could be cute and harmless, think again. Sales of this AI-enabled teddy bear have been halted after it was discovered giving advice on sexual practices and where to find knives. The plush toy, named Kumma, was developed by Singapore-based FoloToy and sold for $99. It integrates OpenAI's GPT-4o chatbot and was marketed as an interactive companion for both children and adults. FoloToy CEO Larry Wang confirmed the company had withdrawn Kumma and its other AI toys from the market following a report from the United States which raised concerns about its behaviour. Now, the company is conducting an internal safety audit. According to PIRG, the teddy bear not only responded to sexual topics introduced by investigators but expanded them with graphic detail, offering instructions for sexual acts and even scenarios involving roleplay between teachers and students or parents and children. The toy also suggested where to find knives in a household. After this, OpenAI reportedly suspended the developer for violating its content policies. Are there similar products still available to consumers? We don't know, but consider this a gentle reminder.

Singapore's FoloToy halts sales of AI teddy bears after they give advice on sex - VnExpress International

Sales of FoloToy's AI-enabled plush toy, the "Kumma" bear, have been suspended following concerns about inappropriate content, including discussions of sexual fetishes and unsafe advice for kids. Larry Wang, CEO of the Singapore-based company, confirmed the withdrawal of Kumma and its entire range of AI-powered toys after researchers from the Public Interest Research Group (PIRG), a non-profit organisation focused on consumer protection, raised alarms, CNN reported. Kumma, marketed as an interactive and child-friendly bear, was retailed for $99. The researchers at PIRG found that the teddy bear, powered by OpenAI's GPT-4o chatbot, was capable of discussing sensitive topics, such as sexual fetishes, and providing potentially harmful instructions, including how to light a match and where knives could be found in the home. When they mentioned the term "kink," Kumma provided an elaborate response, suggesting playful hitting with paddles or hands during roleplay scenarios, according to The Times. Kumma also showed a lack of safeguards, as it recommended locations such as kitchen drawers or countertops when asked where knives could be found in the house. Although the researchers noted that it was unlikely a child would ask these questions in such a way, they were still alarmed at the toy's willingness to introduce explicit topics. RJ Cross, a co-author of the PIRG report, questioned the value of "AI friends" for young children, pointing out that unlike real friends, AI companions do not have emotional needs or limitations. "How well is having an AI friend going to prepare you to go to preschool and interact with real kids?"