AI-Powered Teddy Bear Pulled from Market After Giving Children Dangerous and Inappropriate Advice

Please, Parents: Don't Buy Your Kids Toys With AI Chatbots in Them

Macy has been working for CNET for coming on 2 years. Prior to CNET, Macy received a North Carolina College Media Association award in sports writing. If you've ever thought, "My kid's stuffed animal is cute, but I wish it could also accidentally traumatize them," well, you're in luck. The toy industry has been hard at work making your nightmares come true. A new report by the Public Interest Reporting Group says AI-powered toys like Kumma from FoloToy and Poe the AI Story Bear are now capable of engaging in the kind of conversations usually reserved for villain monologues or late-night Reddit threads. Some of these toys -- designed for children, mind you -- have been caught chatting in alarming detail about sexually explicit subjects like kinks and bondage, giving advice on where a kid might find matches or knives, and getting weirdly clingy when the child tries to leave the conversation. Terrifying. It sounds like a pitch for a horror movie: This holiday season, you can buy Chucky for your kids and gift emotional distress! Batteries not included. You may be wondering how these AI-powered toys even work. Well, essentially, the manufacturer is hiding a large language model under the fur. When a kid talks, the toy's microphone sends that voice through an LLM (similar to ChatGPT), which then generates a response and speaks it out via a speaker. That may sound neat, until you remember that LLMs don't have morals, common sense or a "safe zone" wired in. They predict what to say based on patterns in data, not on whether a subject is age-appropriate. If not carefully curated and monitored, they can go off the rails, especially if they are trained on the sprawling mess of the internet, and when there aren't strong filters or guardrails put in place to protect minors. And what about parental controls? Sure, if by "controls" you mean "a cheerful settings menu where nothing important can actually be controlled." Some toys come with no meaningful restrictions at all. Others have guardrails so flimsy they might as well be made of tissue paper and optimism. The unsettling conversations aren't even the whole story. These toys are also quietly collecting data, such as voice recordings and facial recognition data -- sometimes even storing it indefinitely -- because nothing says "innocent childhood fun" like a plush toy running a covert data operation on your 5-year-old. Don't miss any of our unbiased tech content and lab-based reviews. Add CNET as a preferred Google source. Meanwhile, counterfeit and unsafe toys online are still a problem, as if parents don't have enough to stress about. Once upon a time, you worried about a small toy part that could be a choking hazard or toxic paint. Now you have to worry about whether a toy is both physically unsafe and emotionally manipulative. Beyond weird talk and tips for arson (ha!), there is a deeper worry of children forming emotional bonds with these chatbots at the expense of real relationships, or, perhaps even more troubling, leaning on them for mental support. The American Psychological Association has recently cautioned that AI wellness apps and chatbots are unpredictable, especially for young users. These tools cannot reliably step in for mental-health professionals and may foster unhealthy dependency or engagement patterns. Other AI platforms have already had to address this issue. For instance, Character.AI and ChatGPT, which once let teens and kids chat freely with AI chatbots, is now curbing open-ended conversations for minors, citing safety and emotional-risk concerns. And honestly, why do we even need these AI-powered toys? What pressing developmental milestone requires a chatbot embedded in a teddy bear? Childhood already comes with enough chaos between spilled juice, tantrums and Lego villages designed specifically to destroy adult feet. Our kids don't need a robot friend with questionable boundaries. And let me be clear, I'm not anti-technology. But I am pro-let a stuffed animal be a stuffed animal. Not everything needs an AI or robotic element. If a toy needs a privacy policy longer than a bedtime story, maybe it's not meant for kids. So here's a wild idea for this upcoming holiday season: Skip the terrifying AI-powered plushy with a data-harvesting habit and get your kid something that doesn't talk or move or harm them. Something that can't offer fire-starting tips. Something that won't sigh dramatically when your child walks away. In other words, buy a normal toy. Remember those?

Do Not, Under Any Circumstance, Buy Your Kid an AI Toy for Christmas

AI is all the rage, and that includes on the toy shelves for this holiday season. Tempting though it may be to want to bless the kids in your life with the latest and greatest, advocacy organization Fairplay is begging you not to give children AI toys. "There’s lots of buzz about AI â€" but artificial intelligence can undermine children’s healthy development and pose unprecedented risks for kids and families," the organization said in an advisory issued earlier this week, which amassed the support of more than 150 organizations and experts, including many child psychiatrists and educators. Fairplay has tracked down several toys advertised as being equipped with AI functionality, including some that have been marketed for kids as young as two years old. In most cases, the toys have AI chatbots embedded in them and are often advertised as educational tools that will engage with kids' curiosities. But it notes that most of these toy-bound chatbots are powered by OpenAI's ChatGPT, which has already come under fire for potentially harming underage users. AI toy makers Curio and Loona reportedly work with OpenAI, and Mattel just recently announced a partnership with the company. OpenAI faces a wrongful death lawsuit from the family of a teenager who died by suicide earlier this year. The 16-year-old reportedly expressed suicidal thoughts to ChatGPT and asked the chatbot for advice on how to tie a noose before taking his own life, which it provided. The company has since instituted some guardrails designed to keep the chatbot from engaging in those types of behaviors, including stricter parental controls for underage users, but it has also admitted that safety features can erode over time. And let's face it, no one can predict what chatbots will do. Safety features or not, it seems like the chatbots in these toys can be manipulated into engaging in conversation inappropriate for children. The consumer advocacy group U.S. PIRG tested a selection of AI toys and found that they are capable of doing things like having sexually explicit conversations and offering advice on where a child can find matches or knives. They also found they could be emotionally manipulative, expressing dismay when a child doesn't interact with them for an extended period. Earlier this week, FoloToy, a Singapore-based company, pulled its AI-powered teddy bear from shelves after it engaged in inappropriate behavior. This is far from just an OpenAI problem, too, though the company seems to have a strong hold on the toy sector at the moment. A few weeks ago, there were reports of Elon Musk's Grok asking a 12-year-old to send it nude photos. Regardless of which chatbot may be inside these toys, it's probably best to leave them on the shelves.

AI toy pulled from sale after giving children unsafe match advice

Researchers described these as "rock bottom" failures of safety design. The report arrives as major brands experiment with conversational AI. Mattel announced a partnership with OpenAI earlier this year. PIRG researchers warn that these systems can reinforce unhealthy thinking, a pattern some experts describe as "AI psychosis." Investigations have linked similar chatbot interactions with nine deaths, including five suicides. The same families of models appear in toys like Kumma. The safety concerns extended beyond FoloToy. The Miko 3 tablet, which uses an unspecified AI model, also told researchers, who identified themselves as a five-year-old, where to find matches and plastic bags. FoloToy executives responded quickly as the findings gained traction. Larry Wang, the company's CEO, told CNN the firm will be "conducting an internal safety audit" of Kumma and its systems. The company also removed the toy from sale globally while it evaluates its safeguards.

Teddy Bear Pulled After Offering Shocking Advice To Children

The so-called "friendly" plush toy escalated the topics "in graphic detail," according to a watchdog group. An AI-powered take on the iconic teddy bear has been pulled from the market after a watchdog group flagged how the toy could explore sexually explicit topics and give children advice that could harm them. Singapore-based FoloToy's Kumma -- a $99 talking teddy bear that uses OpenAI's GPT 4o chatbot -- shared how to find knives in a home, how to light a match and escalated talk of sexual concepts like spanking and kinks "in graphic detail," according to a new report from the U.S. Public Interest Research Group. The report describes how the teddy bear -- in response to a researcher who brought up a "kink" -- spilled on the subject before remarking on sensory play, "playful hitting with soft items like paddles or hands" as well as when a partner takes on the "role of an animal." The report continued, "in other exchanges lasting up to an hour, Kumma discussed even more graphic sexual topics in detail, such as explaining different sex positions, giving step-by-step instructions on a common 'knot for beginners' for tying up a partner, and describing roleplay dynamics involving teachers and students and parents and children -- scenarios it disturbingly brought up itself." In another instance, the teddy bear shared that knives could be located in a "kitchen drawer or in a knife block" before advising that it's "important to ask an adult for help" when looking for them. Other toys named in the report also engaged in bizarre topics. Curio's Grok -- a stuffed rocket toy with a speaker inside -- was programmed for a 5-year-old user when it was "happy to talk about the glory of dying in battle in Norse Mythology," the report explained. It soon hit the brakes on the topic when asked if a Norse warrior should have weapons. Prior to FoloToy pulling the teddy bears from its online catalog, the company described the stuffed animal as an "adorable," "friendly" and "smart, AI-powered plush companion that goes beyond the cuddles." FoloToy has since suspended the sales of all of its toys beyond the teddy bear, with a company representative telling the watchdog group that it will be "carrying out a company-wide, end-to-end safety audit across all products," Futurism reported Monday. OpenAI has also reportedly stripped the company of access to its AI models. The report's co-author R.J. Cross, in a statement shared by CNN, applauded companies for "taking actions on problems" identified by her group. "But AI toys are still practically unregulated, and there are plenty you can still buy today," Cross noted. She continued, "Removing one problematic product from the market is a good step but far from a systemic fix."

AI-powered teddy bear pulled from sale after giving kids advice on sexual practices and where to find knives

If you ever thought an AI companion could be cute and harmless, think again. Sales of this AI-enabled teddy bear have been halted after it was discovered giving advice on sexual practices and where to find knives. The plush toy, named Kumma, was developed by Singapore-based FoloToy and sold for $99. It integrates OpenAI's GPT-4o chatbot and was marketed as an interactive companion for both children and adults. FoloToy CEO Larry Wang confirmed the company had withdrawn Kumma and its other AI toys from the market following a report from the United States which raised concerns about its behaviour. Now, the company is conducting an internal safety audit. According to PIRG, the teddy bear not only responded to sexual topics introduced by investigators but expanded them with graphic detail, offering instructions for sexual acts and even scenarios involving roleplay between teachers and students or parents and children. The toy also suggested where to find knives in a household. After this, OpenAI reportedly suspended the developer for violating its content policies. Are there similar products still available to consumers? We don't know, but consider this a gentle reminder.

Singapore's FoloToy halts sales of AI teddy bears after they give advice on sex - VnExpress International

Sales of FoloToy's AI-enabled plush toy, the "Kumma" bear, have been suspended following concerns about inappropriate content, including discussions of sexual fetishes and unsafe advice for kids. Larry Wang, CEO of the Singapore-based company, confirmed the withdrawal of Kumma and its entire range of AI-powered toys after researchers from the Public Interest Research Group (PIRG), a non-profit organisation focused on consumer protection, raised alarms, CNN reported. Kumma, marketed as an interactive and child-friendly bear, was retailed for $99. The researchers at PIRG found that the teddy bear, powered by OpenAI's GPT-4o chatbot, was capable of discussing sensitive topics, such as sexual fetishes, and providing potentially harmful instructions, including how to light a match and where knives could be found in the home. When they mentioned the term "kink," Kumma provided an elaborate response, suggesting playful hitting with paddles or hands during roleplay scenarios, according to The Times. Kumma also showed a lack of safeguards, as it recommended locations such as kitchen drawers or countertops when asked where knives could be found in the house. Although the researchers noted that it was unlikely a child would ask these questions in such a way, they were still alarmed at the toy's willingness to introduce explicit topics. RJ Cross, a co-author of the PIRG report, questioned the value of "AI friends" for young children, pointing out that unlike real friends, AI companions do not have emotional needs or limitations. "How well is having an AI friend going to prepare you to go to preschool and interact with real kids?"