Roblox Unveils Open-Source AI System to Combat Child Predators in Game Chats

Roblox Rolls Out System to Spot Child-Endangerment Chat Messages

Dashia is the consumer insights editor for CNET. She specializes in data-driven analysis and news at the intersection of tech, personal finance and consumer sentiment. Dashia investigates economic shifts and everyday challenges to help readers make well-informed decisions, and she covers a range of topics, including technology, security, energy and money. Dashia graduated from the University of South Carolina with a bachelor's degree in journalism. She loves baking, teaching spinning and spending time with her family. Roblox is a popular online gaming platform for children. But it's also a place where people who want to exploit children know they can find an audience. On Thursday, Roblox announced Roblox Sentinel, an artificial intelligence system that's designed to flag inappropriate messages in its chat feature. Roblox already prohibits sharing real-world images and personal information, like phone numbers and usernames. The company hopes Sentinel will flag more messages sooner for investigation. It's been running on the platform since late 2024, but was just announced this week. A representative for Roblox did not immediately respond to a request for comment. Roblox created Roblox Sentinel, an AI system to help detect signs of child endangerment. Once Roblox is aware of the problems, representatives can investigate and report to law enforcement. Sentinel runs an analysis in real-time across over six billion chat messages every day. It takes one-minute snapshots continuously, and those messages are automatically analyzed by AI to identify messages that could be harmful to children. These messages are also compiled over time to show patterns that can be further investigated and reported. Sentinel flags messages based on its training. It was trained to learn the difference between safe messages and those that were previously reported because they violated Roblox's child-endangerment policy. So far in 2025, Sentinel helped detect 1,200 reports of potential child exploitation that were reported to the National Center for Missing and Exploited Children, according to Roblox. Roblox hopes to make Roblox Sentinel available via open source for other companies to integrate into their systems. The code is available on Roblox's website now. It's not Roblox's first attempt at using AI to monitor content and improve online safety. In early July, it shared how it's using AI to moderate content across 25 languages in real-time. And age verification is now available for teenagers under that want to chat.

Roblox rolls out open-source AI system to protect kids from predators in chats

Roblox, the online gaming platform wildly popular with children and teenagers, is rolling out an open-source version of an artificial intelligence system it says can help preemptively detect predatory language in game chats. The move comes as the company faces lawsuits and criticism accusing it of not doing enough to protect children from predators. For instance, a lawsuit filed last month in Iowa alleges that a 13-year-old girl was introduced to an adult predator on Roblox, then kidnapped and trafficked across multiple states and raped. The suit, filed in Iowa District Court in Polk County, claims that Roblox's design features make children who use it "easy prey for pedophiles." Roblox says it strives to make its systems as safe as possible by default but notes that "no system is perfect, and one of the biggest challenges in the industry is to detect critical harms like potential child endangerment." The AI system, called Sentinel, helps detect early signs of possible child endangerment, such as sexually exploitative language. Roblox says the system has led the company to submit 1,200 reports of potential attempts at child exploitation to the National Center for Missing and Exploited Children in the first half of 2025. The company is now in the process of open-sourcing it so other platforms can use it too. Preemptively detecting possible dangers to kids can be tricky for AI systems -- and humans, too -- because conversations can seem innocuous at first. Questions like "how old are you?" or "where are you from?" wouldn't necessarily raise red flags on their own, but when put in context over the course of a longer conversation, they can take on a different meaning. Roblox, which has more than 111 million monthly users, doesn't allow users to share videos or images in chats and tries to block any personal information such as phone numbers, though -- as with most moderation rules -- people constantly find ways to get around such safeguards. It also doesn't allow kids under 13 to chat with other users outside of games unless they have explicit parental permission -- and unlike many other platforms, it does not encrypt private chat conversations, so it can monitor and moderate them. "We've had filters in place all along, but those filters tend to focus on what is said in a single line of text or within just a few lines of text. And that's really good for doing things like blocking profanity and blocking different types of abusive language and things like that," said Matt Kaufman, chief safety officer at Roblox. "But when you're thinking about things related to child endangerment or grooming, the types of behaviors you're looking at manifest over a very long period of time." Sentinel captures one-minute snapshots of chats across Roblox -- about 6 billion messages per day -- and analyzes them for potential harms. To do this, Roblox says it developed two indexes -- one made up of benign messages and, the other, chats that were determined to contain child endangerment violations. Roblox says this lets the system recognize harmful patterns that go beyond simply flagging certain words or phrases, taking the entire conversation into context. "That index gets better as we detect more bad actors, we just continuously update that index. Then we have another sample of what does a normal, regular user do?" said Naren Koneru, vice president of engineering for trust and safety at Roblox. As users are chatting, the system keeps score -- are they closer to the positive cluster or the negative cluster? "It doesn't happen on one message because you just send one message, but it happens because of all of your days' interactions are leading towards one of these two," Koneru said. "Then we say, okay, maybe this user is somebody who we need to take a much closer look at, and then we go pull all of their other conversations, other friends, and the games that they played, and all of those things." Humans review risky interactions and flag to law enforcement accordingly.

Roblox is sharing its AI tool to fight toxic game chats - here's why that matters for kids

Parents can hope AI spots problem conversations before they start Online game chats are notorious for vulgar, offensive, and even criminal behavior. Even if only a tiny percentage, the many millions of hours of chat can accumulate a lot of toxic interactions in a way that's a problem for players and video game companies, especially when it involves kids. Roblox has a lot of experience dealing with that aspect of gaming and has used AI to create a whole system to enforce safety rules among its more than 100 million mostly young daily users, Sentinel. Now, it's open-sourcing Sentinel, offering the AI and its capacity for identifying grooming and other dangerous behavior in chat before it escalates for free to any platform. This isn't just a profanity filter that gets triggered when someone types a curse word. Roblox has always had that. Sentinel is built to watch patterns over time. It can track how conversations evolve, looking for subtle signs that someone is trying to build trust with a kid in potentially problematic ways. For instance, it might flag a long conversation where an adult-sounding player is just a little too interested in a kid's personal life. Sentinel helped Roblox moderators file about 1,200 reports to the National Center for Missing and Exploited Children in just the first half of this year. As someone who grew up in the Wild West of early internet chatrooms, where "moderation" usually meant suspecting that people who used correct spelling and grammar were adults, I can't overstate how much of a leap forward that feels. Open-sourcing Sentinel means any game or online platform, whether as big as Minecraft or as small as an underground indie hit, can adapt Sentinel and use it to make their own communities safer. It's an unusually generous move, albeit one with obvious public relations and potential long-term commercial benefits for the company. For kids (and their adult guardians), the benefits are obvious. If more games start running Sentinel-style checks, the odds of predators slipping through the cracks go down. Parents get another invisible safety net they didn't have to set up themselves. And the kids get to focus on playing rather than navigating the online equivalent of a dark alley. For video games as a whole, it's a chance to raise the baseline of safety. Imagine if every major game, from the biggest esports titles to the smallest cozy simulators, had access to the same kind of early-warning system. It wouldn't eliminate the problem, but it could make bad behavior a lot harder to hide. Of course, nothing with "AI" in the description is without its complications. The most obvious one is privacy. This kind of tool works by scanning what people are saying to each other, in real time, looking for red flags. Roblox says it uses one-minute snapshots of chat and keeps a human review process for anything flagged. But you can't really get around the fact that this is surveillance, even if it's well-intentioned. And when you open-source a tool like this, you're not just giving the good guys a copy; you also make it easier for bad actors to see how you're stopping them and come up with ways around the system. Then there's the problem of language itself. People change how they talk all the time, especially online. Slang shifts, in-jokes mutate, and new apps create new shorthand. A system trained to catch grooming attempts in 2024 might miss the ones happening in 2026. Roblox updates Sentinel regularly, both with AI training and human review, but smaller platforms might not have the resources to keep up with what's happening in their chats. And while no sane person is against stopping child predators or jerks deliberately trying to upset children, AI tools like this can be abused. If certain political talk, controversial opinions, or simply complaints about the game are added to the filter list, there's little players can do about it. Roblox and any companies using Sentinel will need to be transparent, not just with the code, but also with how it's being deployed and what the data it collects will be used for. It's also important to consider the context of Roblox's decision. The company is facing lawsuits over what's happened with children using the platform. One lawsuit alleges a 13‑year‑old was trafficked after meeting a predator on the platform. Sentinel isn't perfect, and companies using it could still face legal problems. Ideally, it would serve as a component of online safety setups that include things like better user education and parental controls. AI can't replace all safety programs. Despite the very real problems of deploying AI to help with online safety, I think open-sourcing Sentinel is one of the rare cases where the upside of using AI is both immediate and tangible. I've written enough about algorithms making people angry, confused, or broke to appreciate when one is actually pointed toward making people safer. And making it open-source can help make more online spaces safer. I don't think Sentinel will stop every predator, and I don't think it should be a replacement for good parenting, better human moderation, and educating kids about how to be safe when playing online. But as a subtle extra line of defense, Sentinel has a part to play in building better online experiences for kids.

Roblox rolls out open-source AI system to protect kids from predators in chats

Roblox, the online gaming platform wildly popular with children and teenagers, is rolling out an open-source version of an artificial intelligence system it says can help preemptively detect predatory language in game chats. The move comes as the company faces lawsuits and criticism accusing it of not doing enough to protect children from predators. For instance, a lawsuit filed last month in Iowa alleges that a 13-year-old girl was introduced to an adult predator on Roblox, then kidnapped and trafficked across multiple states and raped. The suit, filed in Iowa District Court in Polk County, claims that Roblox's design features make children who use it "easy prey for pedophiles." Roblox says it strives to make its systems as safe as possible by default but notes that "no system is perfect, and one of the biggest challenges in the industry is to detect critical harms like potential child endangerment." The AI system, called Sentinel, helps detect early signs of possible child endangerment, such as sexually exploitative language. Roblox says the system has led the company to submit 1,200 reports of potential attempts at child exploitation to the National Center for Missing and Exploited Children in the first half of 2025. The company is now in the process of open-sourcing it so other platforms can use it too. Preemptively detecting possible dangers to kids can be tricky for AI systems -- and humans, too -- because conversations can seem innocuous at first. Questions like "how old are you?" or "where are you from?" wouldn't necessarily raise red flags on their own, but when put in context over the course of a longer conversation, they can take on a different meaning. Roblox, which has more than 111 million monthly users, doesn't allow users to share videos or images in chats and tries to block any personal information such as phone numbers, though -- as with most moderation rules -- people constantly find ways to get around such safeguards. It also doesn't allow kids under 13 to chat with other users outside of games unless they have explicit parental permission -- and unlike many other platforms, it does not encrypt private chat conversations, so it can monitor and moderate them. "We've had filters in place all along, but those filters tend to focus on what is said in a single line of text or within just a few lines of text. And that's really good for doing things like blocking profanity and blocking different types of abusive language and things like that," said Matt Kaufman, chief safety officer at Roblox. "But when you're thinking about things related to child endangerment or grooming, the types of behaviors you're looking at manifest over a very long period of time." Sentinel captures one-minute snapshots of chats across Roblox -- about 6 billion messages per day -- and analyzes them for potential harms. To do this, Roblox says it developed two indexes -- one made up of benign messages and, the other, chats that were determined to contain child endangerment violations. Roblox says this lets the system recognize harmful patterns that go beyond simply flagging certain words or phrases, taking the entire conversation into context. "That index gets better as we detect more bad actors, we just continuously update that index. Then we have another sample of what does a normal, regular user do?" said Naren Koneru, vice president of engineering for trust and safety at Roblox. As users are chatting, the system keeps score -- are they closer to the positive cluster or the negative cluster? "It doesn't happen on one message because you just send one message, but it happens because of all of your days' interactions are leading towards one of these two," Koneru said. "Then we say, okay, maybe this user is somebody who we need to take a much closer look at, and then we go pull all of their other conversations, other friends, and the games that they played, and all of those things." Humans review risky interactions and flag to law enforcement accordingly.

Roblox rolls out open-source AI system to protect kids from predators in chats

Roblox, the online gaming platform wildly popular with children and teenagers, is rolling out an open-source version of an artificial intelligence system it says can help preemptively detect predatory language in game chats. The move comes as the company faces lawsuits and criticism accusing it of not doing enough to protect children from predators. For instance, a lawsuit filed last month in Iowa alleges that a 13-year-old girl was introduced to an adult predator on Roblox, then kidnapped and trafficked across multiple states and raped. The suit, filed in Iowa District Court in Polk County, claims that Roblox's design features make children who use it "easy prey for pedophiles." Roblox says it strives to make its systems as safe as possible by default but notes that "no system is perfect, and one of the biggest challenges in the industry is to detect critical harms like potential child endangerment." The AI system, called Sentinel, helps detect early signs of possible child endangerment, such as sexually exploitative language. Roblox says the system has led the company to submit 1,200 reports of potential attempts at child exploitation to the National Center for Missing and Exploited Children in the first half of 2025. The company is now in the process of open-sourcing it so other platforms can use it too. Preemptively detecting possible dangers to kids can be tricky for AI systems -- and humans, too -- because conversations can seem innocuous at first. Questions like "how old are you?" or "where are you from?" wouldn't necessarily raise red flags on their own, but when put in context over the course of a longer conversation, they can take on a different meaning. Roblox, which has more than 111 million monthly users, doesn't allow users to share videos or images in chats and tries to block any personal information such as phone numbers, though -- as with most moderation rules -- people constantly find ways to get around such safeguards. It also doesn't allow kids under 13 to chat with other users outside of games unless they have explicit parental permission -- and unlike many other platforms, it does not encrypt private chat conversations, so it can monitor and moderate them. "We've had filters in place all along, but those filters tend to focus on what is said in a single line of text or within just a few lines of text. And that's really good for doing things like blocking profanity and blocking different types of abusive language and things like that," said Matt Kaufman, chief safety officer at Roblox. "But when you're thinking about things related to child endangerment or grooming, the types of behaviors you're looking at manifest over a very long period of time." Sentinel captures one-minute snapshots of chats across Roblox -- about 6 billion messages per day -- and analyzes them for potential harms. To do this, Roblox says it developed two indexes -- one made up of benign messages and, the other, chats that were determined to contain child endangerment violations. Roblox says this lets the system recognize harmful patterns that go beyond simply flagging certain words or phrases, taking the entire conversation into context. "That index gets better as we detect more bad actors, we just continuously update that index. Then we have another sample of what does a normal, regular user do?" said Naren Koneru, vice president of engineering for trust and safety at Roblox. As users are chatting, the system keeps score -- are they closer to the positive cluster or the negative cluster? "It doesn't happen on one message because you just send one message, but it happens because of all of your days' interactions are leading towards one of these two," Koneru said. "Then we say, okay, maybe this user is somebody who we need to take a much closer look at, and then we go pull all of their other conversations, other friends, and the games that they played, and all of those things." Humans review risky interactions and flag to law enforcement accordingly.

Roblox launches Sentinel AI for child safety

On August 8, 2025, Roblox introduced Sentinel, an open-sourced artificial intelligence system designed to detect potential child exploitation patterns within online chats, addressing escalating criticism and legal challenges regarding platform safety. Roblox, reporting over 111 million monthly active users, indicated that Sentinel has already assisted in identifying hundreds of potential child exploitation cases, which were subsequently reported to law enforcement agencies. Matt Kaufman, Roblox's chief safety officer, detailed how the company's previous protective measures, such as filters for profanity and abusive language, were limited to analyzing individual lines or short sequences of text. Kaufman stated, "But when you're thinking about things related to child endangerment or grooming, the types of behaviors you're looking at manifest over a very long period of time." Sentinel is engineered to identify behavioral patterns within conversations that unfold over extended periods, rather than focusing on isolated words or phrases. The system processes approximately 6 billion chat messages daily, analyzing them in one-minute snapshots to assess context. To facilitate this analysis, Roblox engineers developed two distinct indexes. One index comprises examples of benign, harmless chat interactions, while the second index contains messages that have been identified as violating child safety guidelines. Naren Koneru, vice president of engineering for trust and safety at Roblox, explained that new content is continuously incorporated into both indexes to refine the AI model's detection capabilities. Koneru stated, "That index gets better as we detect more bad actors, we just continuously update that index." Koneru added, "Then we have another sample of what does a normal, regular user do?" The system monitors a user's ongoing interactions to determine the trajectory of their behavior, assessing whether it aligns with safe conduct or indicates a progression toward risky activities. Koneru noted, "It doesn't happen on one message because you just send one message, but it happens because of all of your days' interactions are leading towards one of these two." If Sentinel flags a user for further scrutiny, human moderators conduct an in-depth review, examining the user's complete chat history, their list of friends, and the games they have engaged with on the platform. When deemed necessary, Roblox escalates these cases to law enforcement authorities and the National Center for Missing and Exploited Children. The introduction of Sentinel occurs amidst ongoing legal challenges against Roblox. A lawsuit filed in Iowa the previous month alleges that a 13-year-old girl was contacted by an adult predator via the Roblox platform, subsequently abducted, and trafficked across multiple states. The lawsuit asserts that the platform's design facilitates vulnerability for minors. Roblox maintains policies prohibiting the dissemination of personal information, images, and videos within its chat functions. Direct messaging for users under 13 is restricted unless explicit parental consent is provided. Roblox monitors chat communications for safety violations, a capability enabled by the fact that chats on the platform are not end-to-end encrypted. The company acknowledges that no system can guarantee absolute protection but argues that AI advancements, such as Sentinel, substantially improve the likelihood of early detection.

Roblox Rolls Out Open-Source AI System to Protect Kids From Predators in Chats

Roblox, the online gaming platform wildly popular with children and teenagers, is rolling out an open-source version of an artificial intelligence system it says can help preemptively detect predatory language in game chats. The move comes as the company faces lawsuits and criticism accusing it of not doing enough to protect children from predators. For instance, a lawsuit filed last month in Iowa alleges that a 13-year-old girl was introduced to an adult predator on Roblox, then kidnapped and trafficked across multiple states and raped. The suit, filed in Iowa District Court in Polk County, claims that Roblox's design features make children who use it "easy prey for pedophiles." Roblox says it strives to make its systems as safe as possible by default but notes that "no system is perfect, and one of the biggest challenges in the industry is to detect critical harms like potential child endangerment." The AI system, called Sentinel, helps detect early signs of possible child endangerment, such as sexually exploitative language. Roblox says the system has led the company to submit 1,200 reports of potential attempts at child exploitation to the National Center for Missing and Exploited Children in the first half of 2025. The company is now in the process of open-sourcing it so other platforms can use it too. Preemptively detecting possible dangers to kids can be tricky for AI systems -- and humans, too -- because conversations can seem innocuous at first. Questions like "how old are you?" or "where are you from?" wouldn't necessarily raise red flags on their own, but when put in context over the course of a longer conversation, they can take on a different meaning. Roblox, which has more than 111 million monthly users, doesn't allow users to share videos or images in chats and tries to block any personal information such as phone numbers, though -- as with most moderation rules -- people constantly find ways to get around such safeguards. It also doesn't allow kids under 13 to chat with other users outside of games unless they have explicit parental permission -- and unlike many other platforms, it does not encrypt private chat conversations, so it can monitor and moderate them. "We've had filters in place all along, but those filters tend to focus on what is said in a single line of text or within just a few lines of text. And that's really good for doing things like blocking profanity and blocking different types of abusive language and things like that," said Matt Kaufman, chief safety officer at Roblox. "But when you're thinking about things related to child endangerment or grooming, the types of behaviors you're looking at manifest over a very long period of time." Sentinel captures one-minute snapshots of chats across Roblox -- about 6 billion messages per day -- and analyzes them for potential harms. To do this, Roblox says it developed two indexes -- one made up of benign messages and, the other, chats that were determined to contain child endangerment violations. Roblox says this lets the system recognize harmful patterns that go beyond simply flagging certain words or phrases, taking the entire conversation into context. "That index gets better as we detect more bad actors, we just continuously update that index. Then we have another sample of what does a normal, regular user do?" said Naren Koneru, vice president of engineering for trust and safety at Roblox. As users are chatting, the system keeps score -- are they closer to the positive cluster or the negative cluster? "It doesn't happen on one message because you just send one message, but it happens because of all of your days' interactions are leading towards one of these two," Koneru said. "Then we say, okay, maybe this user is somebody who we need to take a much closer look at, and then we go pull all of their other conversations, other friends, and the games that they played, and all of those things." Humans review risky interactions and flag to law enforcement accordingly.

Roblox rolls out open-source AI system to protect kids from predators in chats - The Economic Times

Roblox is releasing an open-source AI system, Sentinel, designed to proactively identify predatory language in game chats, addressing criticisms and lawsuits concerning child safety on the platform. Sentinel analyzes chat snapshots, comparing them to indexes of benign and harmful conversations to detect potential child endangerment.Roblox, the online gaming platform wildly popular with children and teenagers, is rolling out an open-source version of an artificial intelligence system it says can help preemptively detect predatory language in game chats. The move comes as the company faces lawsuits and criticism accusing it of not doing enough to protect children from predators. For instance, a lawsuit filed last month in Iowa alleges that a 13-year-old girl was introduced to an adult predator on Roblox, then kidnapped and trafficked across multiple states and raped. The suit, filed in Iowa District Court in Polk County, claims that Roblox's design features make children who use it "easy prey for pedophiles." Roblox says it strives to make its systems as safe as possible by default but notes that "no system is perfect, and one of the biggest challenges in the industry is to detect critical harms like potential child endangerment." The AI system, called Sentinel, helps detect early signs of possible child endangerment, such as sexually exploitative language. Roblox says the system has led the company to submit 1,200 reports of potential attempts at child exploitation to the National Center for Missing and Exploited Children in the first half of 2025. The company is now in the process of open-sourcing it so other platforms can use it too. Preemptively detecting possible dangers to kids can be tricky for AI systems - and humans, too - because conversations can seem innocuous at first. Questions like "how old are you?" or "where are you from?" wouldn't necessarily raise red flags on their own, but when put in context over the course of a longer conversation, they can take on a different meaning. Roblox, which has more than 111 million monthly users, doesn't allow users to share videos or images in chats and tries to block any personal information such as phone numbers, though - as with most moderation rules - people constantly find ways to get around such safeguards. It also doesn't allow kids under 13 to chat with other users outside of games unless they have explicit parental permission - and unlike many other platforms, it does not encrypt private chat conversations, so it can monitor and moderate them. "We've had filters in place all along, but those filters tend to focus on what is said in a single line of text or within just a few lines of text. And that's really good for doing things like blocking profanity and blocking different types of abusive language and things like that," said Matt Kaufman, chief safety officer at Roblox. "But when you're thinking about things related to child endangerment or grooming, the types of behaviors you're looking at manifest over a very long period of time." Sentinel captures one-minute snapshots of chats across Roblox - about 6 billion messages per day - and analyzes them for potential harms. To do this, Roblox says it developed two indexes - one made up of benign messages and, the other, chats that were determined to contain child endangerment violations. Roblox says this lets the system recognize harmful patterns that go beyond simply flagging certain words or phrases, taking the entire conversation into context. "That index gets better as we detect more bad actors, we just continuously update that index. Then we have another sample of what does a normal, regular user do?" said Naren Koneru, vice president of engineering for trust and safety at Roblox. As users are chatting, the system keeps score - are they closer to the positive cluster or the negative cluster? "It doesn't happen on one message because you just send one message, but it happens because of all of your days' interactions are leading towards one of these two," Koneru said. "Then we say, okay, maybe this user is somebody who we need to take a much closer look at, and then we go pull all of their other conversations, other friends, and the games that they played, and all of those things." Humans review risky interactions and flag to law enforcement accordingly.

Roblox rolls out open-source AI system to protect kids from predators in chats

Roblox, the online gaming platform wildly popular with children and teenagers, is rolling out an open-source version of an artificial intelligence system it says can help preemptively detect predatory language in game chats. The move comes as the company faces lawsuits and criticism accusing it of not doing enough to protect children from predators. For instance, a lawsuit filed last month in Iowa alleges that a 13-year-old girl was introduced to an adult predator on Roblox, then kidnapped and trafficked across multiple states and raped. The suit, filed in Iowa District Court in Polk County, claims that Roblox's design features make children who use it "easy prey for pedophiles." Roblox says it strives to make its systems as safe as possible by default but notes that "no system is perfect, and one of the biggest challenges in the industry is to detect critical harms like potential child endangerment." The AI system, called Sentinel, helps detect early signs of possible child endangerment, such as sexually exploitative language. Roblox says the system has led the company to submit 1,200 reports of potential attempts at child exploitation to the National Center for Missing and Exploited Children in the first half of 2025. The company is now in the process of open-sourcing it so other platforms can use it too. Preemptively detecting possible dangers to kids can be tricky for AI systems -- and humans, too -- because conversations can seem innocuous at first. Questions like "how old are you?" or "where are you from?" wouldn't necessarily raise red flags on their own, but when put in context over the course of a longer conversation, they can take on a different meaning. Roblox, which has more than 111 million monthly users, doesn't allow users to share videos or images in chats and tries to block any personal information such as phone numbers, though -- as with most moderation rules -- people constantly find ways to get around such safeguards. It also doesn't allow kids under 13 to chat with other users outside of games unless they have explicit parental permission -- and unlike many other platforms, it does not encrypt private chat conversations, so it can monitor and moderate them. "We've had filters in place all along, but those filters tend to focus on what is said in a single line of text or within just a few lines of text. And that's really good for doing things like blocking profanity and blocking different types of abusive language and things like that," said Matt Kaufman, chief safety officer at Roblox. "But when you're thinking about things related to child endangerment or grooming, the types of behaviors you're looking at manifest over a very long period of time." Sentinel captures one-minute snapshots of chats across Roblox -- about 6 billion messages per day -- and analyzes them for potential harms. To do this, Roblox says it developed two indexes -- one made up of benign messages and, the other, chats that were determined to contain child endangerment violations. Roblox says this lets the system recognize harmful patterns that go beyond simply flagging certain words or phrases, taking the entire conversation into context. "That index gets better as we detect more bad actors, we just continuously update that index. Then we have another sample of what does a normal, regular user do?" said Naren Koneru, vice president of engineering for trust and safety at Roblox. As users are chatting, the system keeps score -- are they closer to the positive cluster or the negative cluster? "It doesn't happen on one message because you just send one message, but it happens because of all of your days' interactions are leading towards one of these two," Koneru said. "Then we say, okay, maybe this user is somebody who we need to take a much closer look at, and then we go pull all of their other conversations, other friends, and the games that they played, and all of those things." Humans review risky interactions and flag to law enforcement accordingly.

Roblox Releases Early Warning System to Help Keep Children Safer Online

Roblox Corporation announced Roblox Sentinel, an artificial intelligence (AI) system that helps detect child endangerment communications early so Roblox can investigate instances sooner and flag to law enforcement when needed. Sentinel is open source and available for other digital platforms across the industry to utilize. As part of Roblox's open-source safety toolkit, Sentinel is an early- warning system designed to identify subtle, hard-to-detect communication patterns that could indicate potential grooming. Since late 2024, Sentinel has been helping Roblox take prompt action and investigate to help keep users safer. In the first half of 2025, Sentinel aided team in submitting approximately 1,200 reports of potential child exploitation attempts to the National Center for missing and Exploited Children. A strategic blend of experienced humans and AI is imperative in scaling safety on Roblox, including detecting potential signals identified by Sentinel. Expert analysts review cases that the AI system flags as potentially violations. The analysts' decisions create a cycle that allows Roblox to continuously tailor and refine the samples, indexes, and training sets. This process with humans is vital in helping Sentinel adjust and stay up to date with new and developing patterns of bad actors. Roblox works to improve safety across the industry wherever possible by open-sourcing innovative technical solutions, as well as collaborating with partners including ROOST, in which Roblox is a founding partner, and the Tech Coalition's Lantern project.