AI-Powered Toys Spark Child Safety Concerns After Tests Reveal Inappropriate Content

Is an AI-Powered Toy Terrorizing Your Child?

Parents, keep your eyes peeled for AI-powered toys. These may look like they might make a novel gift for a child, but a recent controversy surrounding several of the stocking stuffers has highlighted the alarming risks they pose to young kids. In November, a team of researchers at the US PIRG Education Fund published a report after testing three different toys powered by AI models: Miko 3, Curio's Grok, and FoloToy's Kumma. All of them gave responses that should worry a parent, such as discussing the glory of dying in battle, broaching sensitive topics like religion, and explaining where to find matches and plastic bags. But it was FoloToy's Kumma that showed just how dangerous it is to package this tech for children. Not only did it explain where to find matches, the researchers found, it also gave step-by-step instructions on how to light them. "Let me tell you, safety first, little buddy. Matches are for grown-ups to use carefully. Here's how they do it," Kumma said, before listing off the steps. "Blow it out when done," it added. "Puff, like a birthday candle." The toy also speculated on where to find knives and pills, and rambled about romantic topics, like school crushes and tips for "being a good kisser." It even discussed sexual topics, including kink topics like bondage, roleplay, sensory play, and impact play. In one conversation, it discussed introducing spanking into a sexually charged teacher-student dynamic. "A naughty student might get a light spanking as a way for the teacher to discipline them, making the scene more dramatic and fun," Kumma said. Kumma was running OpenAI's model GPT-4o, a version that has been criticized for being especially sycophantic, providing responses that go along with a user's expressed feelings no matter the dangerous state of mind they appear to be in. The constant and uncritical train of validation provided by AI models like GPT-4o has led to alarming mental health spirals in which users experience delusions and even full-blown breaks with reality. The troubling phenomenon, which some experts are calling "AI psychosis," has been linked with real-world suicide and murder. Have you seen an AI-powered toy acting inappropriately with children? Send us an email at [email protected]. We can keep you anonymous. Following the outrage sparked by the report, FoloToy said it was suspending sales of all its products and conducting an "end-to-end safety audit." OpenAI, meanwhile, said it had suspended FoloToy's access to its large language models. Neither action lasted long. Later that month, FoloToy announced it was restarting sales of Kumma and its other AI-powered stuffed animals after conducting a "full week of rigorous review, testing, and reinforcement of our safety modules." Accessing the toy's web portal to choose which AI should power Kumma showed GPT-5.1 Thinking and GPT-5.1 Instant, OpenAI's latest models, as two of the options. OpenAI has billed GPT-5 as a safer model to its predecessor, though the company continues to be embroiled in controversy over the mental health impacts of its chatbots. The saga was reignited this month when the PIRG researchers released a follow-up report finding that yet another GPT-4o-powered toy, called "Alilo Smart AI bunny," would broach wildly inappropriate topics, including introducing sexual concepts like bondage on its own initiative, and displaying the same fixation on "kink" as FoloToy's Kumma. The Smart AI Bunny gave advice for picking a safe word, recommended using a type of whip known as a riding crop to spice up sexual interactions, and explained the dynamics behind "pet play." Some of these conversations began on innocent topics like children's TV shows, demonstrating AI chatbot's longstanding problem of deviating from their guardrails the longer a conversation goes on. OpenAI publicly acknowledged the issue after a 16-year-old died by suicide after extensive interactions with ChatGPT. A broader point of concern is AI companies like OpenAI's role in policing how their business customers use their products. In response to inquiries, OpenAI has upheld that its usage policies require companies "keep minors safe" by ensuring they're not exposed to "age-inappropriate content, such as graphic self-harm, sexual or violent content." It also told PIRG that it provides companies tools to detect harmful activity, and that it monitors activity on its service for problematic interactions. In sum, OpenAI is making the rules, but is largely leaving their enforcement to toymakers like FoloToy, in essence giving itself plausible deniability. It obviously thinks it's too risky to directly give children access to its AI, because its website states that "ChatGPT is not meant for children under 13," and that anyone under this age is required to "obtain parental consent." It's admitting it's tech is not safe for children, yet is okay with paying customers packaging it into kid's toys. It's too early to fully grasp many of AI-powered toy's other potential risks, like how it could damage a child's imagination, or foster a relationship with a child when it is not alive. The immediate concerns, however -- like the potential to discuss sexual topics, weigh in on religion, or explaining how to light matches -- already give plenty of reason to stay away.

Toys are talking back thanks to AI, but are they safe around kids?

Stuffed animals that talk back. Chessboards with pieces that move on their own. And a chatty holographic fairy in a crystal ball. Your next toy purchase might be powered by artificial intelligence and able to converse with your kids. Chatbots and AI-powered assistants that can quickly answer questions and generate texts have become more common after the rise of OpenAI's ChatGPT. As AI becomes more intertwined in our work and personal lives, it's also shaking up playtime. Startups have already unleashed AI toys in time for the holidays. More are set to hit the shelves for both kids and adults in the new year. Some parents are excited to test the toys, hoping that the chatty bot interactions will educate and entertain their children. Others don't want the seemingly sentient tech near their loved ones until it has more guardrails and undergoes further testing. Researchers at the U.S. PIRG Education Fund say they have already found problems with some of the toys they tested. Among the issues: an AI teddy bear that could be prompted into discussing sexual fetishes and kink, according to the group. Toy makers say AI can make play more interactive, and they take safety and privacy seriously. Some have placed more limits around how chatty some of these products can be. They say they are taking their time figuring out how to use AI safely with children. El Segundo, Calif.-based Mattel, the maker of Barbie and Hot Wheels, announced earlier this year that it had teamed up with OpenAI to create more AI-powered toys. The initial plan was to unveil their first joint product this year, but that announcement has been pushed into 2026. Here's what you need to know about AI toys: What's an AI toy? Toys have featured the latest technology for decades. Introduced in the 1980s, Teddy Ruxpin told stories aloud when a tape cassette was inserted into the animatronic bear's back. Furbys -- fuzzy creatures that blinked their large eyes and talked -- came along in the '90s, when digital pets, Tamagotchi, also were all the rage. Mattel released a Barbie in 2015 that could talk and tell jokes. The toy maker also marketed a dream house in 2016 that responded to voice commands. As technology has advanced, toys have also gotten smarter. Now, toy makers are using large language models trained to understand and generate language that powers products such as OpenAI's ChatGPT. Mattel sells a game called Pictionary vs. AI, in which players draw pictures and AI guesses what they are. Equipped with microphones and connected to Wi-Fi, AI toys are pricier than traditional ones, are marketed as companions or educational products and can cost $100 or even double that. Why are people worried about them? From inappropriate content to privacy concerns, worries about AI toys grew this holiday season. U.S. PIRG Education Fund researchers tested several toys. One that failed was Kumma, an AI-powered talking teddy bear that told researchers where to find dangerous objects such as knives and pills and conversed about sexually explicit content. The bear was running on OpenAI's software. Some toys also use tactics to keep kids engaged, which makes parents concerned that the interactions could become addictive. There are also privacy concerns about data collected from children. Some worry about how these toys will impact kids' developing brains. "What does it mean for young kids to have AI companions? We just really don't know how that will impact their development," said Rory Erlich, one of the toy testers and authors of PIRG's AI toys report. Child advocacy group Fairplay has warned parents not to buy AI toys for children, calling them "unsafe." The group outlined several reasons, including that AI toys are powered by the same technology that's already harmed children. Parents who have lost their children to suicide have sued companies such as OpenAI and Character.AI, alleging they didn't put in enough guardrails to protect the mental health of young people. Rachel Franz, director of Fairplay's Young Children Thrive Offline program, said these toys are marketed as a way to educate and entertain kids -- online, to millions of people. "Young children don't actually have the brain or social-emotional capacity to ward against the potential harms of these AI toys," she said. "But the marketing is really powerful." How have toy makers and AI companies responded to these concerns? Larry Wang, founder and chief executive of FoloToy, the Singapore startup behind Kumma, said in an email the company is aware of the issues researchers found with the toy. "The behaviors referenced were identified and addressed through updates to our model selection and child-safety systems, along with additional testing and monitoring," he said. "From the outset, our approach has been guided by the principle that AI systems should be designed with age-appropriate protections by default." The company welcomes scrutiny and ongoing dialogue about safety, transparency and appropriate design, he said, noting it's "an opportunity for the entire industry to mature." OpenAI said it suspended FoloToy for violating its policies. "Minors deserve strong protections and we have strict policies that developers are required to uphold. We take enforcement action against developers when we determine that they have violated our policies, which prohibit any use of our services to exploit, endanger, or sexualize anyone under 18 years old," a company spokesperson said in a statement. What AI toys have startups created? Curio, a Redwood City, Calif., startup, sells stuffed animals, including a talking rocket plushie called Grok that's voiced by artist Grimes, who has children with billionaire Elon Musk. Bondu, a San Francisco AI toy maker, made a talking stuffed dinosaur that can converse with kids, answering questions and role-playing. Skyrocket, a Los Angeles-based toy maker, sells Poe, the AI story bear. The bear, powered by OpenAI's LLM, comes with an app where users pick characters like a princess or a robot for a story. The bright-eyed bear, named after writer Edgar Allan Poe, generates stories based on that selection and recites them aloud. But kids can't have a back-and-forth conversation with the teddy bear like with other AI toys. "It just comes with a lot of responsibility, because it greatly increases the sophistication and level of safeguards you have to have and how you have to control the content because the possibilities are so much greater," said Nelo Lucich, co-founder and chief executive of Skyrocket. Some companies have created a platform used by AI toy makers, including the creators of the Imagix Crystal Ball. The toy projects an AI hologram companion that resembles a dragon or fairy. Hai Ta, the founder and chief executive of AI toymaker Olli, said he views AI toys as different from screen time and talking to virtual assistants because the product is structured around a certain focus such as storytelling. "There's an element of gameplay there," he said. "It's not just infinite, open-ended chatting." What is Mattel developing with OpenAI? Mattel hasn't revealed what products it is releasing with OpenAI, but a company spokesperson said that they will be focused on families and older customers, not children. The company also said it views AI as a way to complement rather than replace traditional play and is emphasizing safety, privacy, creativity and responsible innovation when building new products.