Roblox Unveils AI-Powered 'Sentinel' System to Combat Child Exploitation in Game Chats

Roblox Rolls Out System to Spot Child-Endangerment Chat Messages

Dashia is the consumer insights editor for CNET. She specializes in data-driven analysis and news at the intersection of tech, personal finance and consumer sentiment. Dashia investigates economic shifts and everyday challenges to help readers make well-informed decisions, and she covers a range of topics, including technology, security, energy and money. Dashia graduated from the University of South Carolina with a bachelor's degree in journalism. She loves baking, teaching spinning and spending time with her family. Roblox is a popular online gaming platform for children. But it's also a place where people who want to exploit children know they can find an audience. On Thursday, Roblox announced Roblox Sentinel, an artificial intelligence system that's designed to flag inappropriate messages in its chat feature. Roblox already prohibits sharing real-world images and personal information, like phone numbers and usernames. The company hopes Sentinel will flag more messages sooner for investigation. It's been running on the platform since late 2024, but was just announced this week. A representative for Roblox did not immediately respond to a request for comment. Roblox created Roblox Sentinel, an AI system to help detect signs of child endangerment. Once Roblox is aware of the problems, representatives can investigate and report to law enforcement. Sentinel runs an analysis in real-time across over six billion chat messages every day. It takes one-minute snapshots continuously, and those messages are automatically analyzed by AI to identify messages that could be harmful to children. These messages are also compiled over time to show patterns that can be further investigated and reported. Sentinel flags messages based on its training. It was trained to learn the difference between safe messages and those that were previously reported because they violated Roblox's child-endangerment policy. So far in 2025, Sentinel helped detect 1,200 reports of potential child exploitation that were reported to the National Center for Missing and Exploited Children, according to Roblox. Roblox hopes to make Roblox Sentinel available via open source for other companies to integrate into their systems. The code is available on Roblox's website now. It's not Roblox's first attempt at using AI to monitor content and improve online safety. In early July, it shared how it's using AI to moderate content across 25 languages in real-time. And age verification is now available for teenagers under that want to chat.

Roblox rolls out open-source AI system to protect kids from predators in chats

Roblox, the online gaming platform wildly popular with children and teenagers, is rolling out an open-source version of an artificial intelligence system it says can help preemptively detect predatory language in game chats. The move comes as the company faces lawsuits and criticism accusing it of not doing enough to protect children from predators. For instance, a lawsuit filed last month in Iowa alleges that a 13-year-old girl was introduced to an adult predator on Roblox, then kidnapped and trafficked across multiple states and raped. The suit, filed in Iowa District Court in Polk County, claims that Roblox's design features make children who use it "easy prey for pedophiles." Roblox says it strives to make its systems as safe as possible by default but notes that "no system is perfect, and one of the biggest challenges in the industry is to detect critical harms like potential child endangerment." The AI system, called Sentinel, helps detect early signs of possible child endangerment, such as sexually exploitative language. Roblox says the system has led the company to submit 1,200 reports of potential attempts at child exploitation to the National Center for Missing and Exploited Children in the first half of 2025. The company is now in the process of open-sourcing it so other platforms can use it too. Preemptively detecting possible dangers to kids can be tricky for AI systems -- and humans, too -- because conversations can seem innocuous at first. Questions like "how old are you?" or "where are you from?" wouldn't necessarily raise red flags on their own, but when put in context over the course of a longer conversation, they can take on a different meaning. Roblox, which has more than 111 million monthly users, doesn't allow users to share videos or images in chats and tries to block any personal information such as phone numbers, though -- as with most moderation rules -- people constantly find ways to get around such safeguards. It also doesn't allow kids under 13 to chat with other users outside of games unless they have explicit parental permission -- and unlike many other platforms, it does not encrypt private chat conversations, so it can monitor and moderate them. "We've had filters in place all along, but those filters tend to focus on what is said in a single line of text or within just a few lines of text. And that's really good for doing things like blocking profanity and blocking different types of abusive language and things like that," said Matt Kaufman, chief safety officer at Roblox. "But when you're thinking about things related to child endangerment or grooming, the types of behaviors you're looking at manifest over a very long period of time." Sentinel captures one-minute snapshots of chats across Roblox -- about 6 billion messages per day -- and analyzes them for potential harms. To do this, Roblox says it developed two indexes -- one made up of benign messages and, the other, chats that were determined to contain child endangerment violations. Roblox says this lets the system recognize harmful patterns that go beyond simply flagging certain words or phrases, taking the entire conversation into context. "That index gets better as we detect more bad actors, we just continuously update that index. Then we have another sample of what does a normal, regular user do?" said Naren Koneru, vice president of engineering for trust and safety at Roblox. As users are chatting, the system keeps score -- are they closer to the positive cluster or the negative cluster? "It doesn't happen on one message because you just send one message, but it happens because of all of your days' interactions are leading towards one of these two," Koneru said. "Then we say, okay, maybe this user is somebody who we need to take a much closer look at, and then we go pull all of their other conversations, other friends, and the games that they played, and all of those things." Humans review risky interactions and flag to law enforcement accordingly.

Roblox rolls out open-source AI system to protect kids from predators in chats

Roblox, the online gaming platform wildly popular with children and teenagers, is rolling out an open-source version of an artificial intelligence system it says can help preemptively detect predatory language in game chats. The move comes as the company faces lawsuits and criticism accusing it of not doing enough to protect children from predators. For instance, a lawsuit filed last month in Iowa alleges that a 13-year-old girl was introduced to an adult predator on Roblox, then kidnapped and trafficked across multiple states and raped. The suit, filed in Iowa District Court in Polk County, claims that Roblox's design features make children who use it "easy prey for pedophiles." Roblox says it strives to make its systems as safe as possible by default but notes that "no system is perfect, and one of the biggest challenges in the industry is to detect critical harms like potential child endangerment." The AI system, called Sentinel, helps detect early signs of possible child endangerment, such as sexually exploitative language. Roblox says the system has led the company to submit 1,200 reports of potential attempts at child exploitation to the National Center for Missing and Exploited Children in the first half of 2025. The company is now in the process of open-sourcing it so other platforms can use it too. Preemptively detecting possible dangers to kids can be tricky for AI systems -- and humans, too -- because conversations can seem innocuous at first. Questions like "how old are you?" or "where are you from?" wouldn't necessarily raise red flags on their own, but when put in context over the course of a longer conversation, they can take on a different meaning. Roblox, which has more than 111 million monthly users, doesn't allow users to share videos or images in chats and tries to block any personal information such as phone numbers, though -- as with most moderation rules -- people constantly find ways to get around such safeguards. It also doesn't allow kids under 13 to chat with other users outside of games unless they have explicit parental permission -- and unlike many other platforms, it does not encrypt private chat conversations, so it can monitor and moderate them. "We've had filters in place all along, but those filters tend to focus on what is said in a single line of text or within just a few lines of text. And that's really good for doing things like blocking profanity and blocking different types of abusive language and things like that," said Matt Kaufman, chief safety officer at Roblox. "But when you're thinking about things related to child endangerment or grooming, the types of behaviors you're looking at manifest over a very long period of time." Sentinel captures one-minute snapshots of chats across Roblox -- about 6 billion messages per day -- and analyzes them for potential harms. To do this, Roblox says it developed two indexes -- one made up of benign messages and, the other, chats that were determined to contain child endangerment violations. Roblox says this lets the system recognize harmful patterns that go beyond simply flagging certain words or phrases, taking the entire conversation into context. "That index gets better as we detect more bad actors, we just continuously update that index. Then we have another sample of what does a normal, regular user do?" said Naren Koneru, vice president of engineering for trust and safety at Roblox. As users are chatting, the system keeps score -- are they closer to the positive cluster or the negative cluster? "It doesn't happen on one message because you just send one message, but it happens because of all of your days' interactions are leading towards one of these two," Koneru said. "Then we say, okay, maybe this user is somebody who we need to take a much closer look at, and then we go pull all of their other conversations, other friends, and the games that they played, and all of those things." Humans review risky interactions and flag to law enforcement accordingly.

Roblox rolls out open-source AI system to protect kids from predators in chats

Roblox, the online gaming platform wildly popular with children and teenagers, is rolling out an open-source version of an artificial intelligence system it says can help preemptively detect predatory language in game chats. The move comes as the company faces lawsuits and criticism accusing it of not doing enough to protect children from predators. For instance, a lawsuit filed last month in Iowa alleges that a 13-year-old girl was introduced to an adult predator on Roblox, then kidnapped and trafficked across multiple states and raped. The suit, filed in Iowa District Court in Polk County, claims that Roblox's design features make children who use it "easy prey for pedophiles." Roblox says it strives to make its systems as safe as possible by default but notes that "no system is perfect, and one of the biggest challenges in the industry is to detect critical harms like potential child endangerment." The AI system, called Sentinel, helps detect early signs of possible child endangerment, such as sexually exploitative language. Roblox says the system has led the company to submit 1,200 reports of potential attempts at child exploitation to the National Center for Missing and Exploited Children in the first half of 2025. The company is now in the process of open-sourcing it so other platforms can use it too. Preemptively detecting possible dangers to kids can be tricky for AI systems -- and humans, too -- because conversations can seem innocuous at first. Questions like "how old are you?" or "where are you from?" wouldn't necessarily raise red flags on their own, but when put in context over the course of a longer conversation, they can take on a different meaning. Roblox, which has more than 111 million monthly users, doesn't allow users to share videos or images in chats and tries to block any personal information such as phone numbers, though -- as with most moderation rules -- people constantly find ways to get around such safeguards. It also doesn't allow kids under 13 to chat with other users outside of games unless they have explicit parental permission -- and unlike many other platforms, it does not encrypt private chat conversations, so it can monitor and moderate them. "We've had filters in place all along, but those filters tend to focus on what is said in a single line of text or within just a few lines of text. And that's really good for doing things like blocking profanity and blocking different types of abusive language and things like that," said Matt Kaufman, chief safety officer at Roblox. "But when you're thinking about things related to child endangerment or grooming, the types of behaviors you're looking at manifest over a very long period of time." Sentinel captures one-minute snapshots of chats across Roblox -- about 6 billion messages per day -- and analyzes them for potential harms. To do this, Roblox says it developed two indexes -- one made up of benign messages and, the other, chats that were determined to contain child endangerment violations. Roblox says this lets the system recognize harmful patterns that go beyond simply flagging certain words or phrases, taking the entire conversation into context. "That index gets better as we detect more bad actors, we just continuously update that index. Then we have another sample of what does a normal, regular user do?" said Naren Koneru, vice president of engineering for trust and safety at Roblox. As users are chatting, the system keeps score -- are they closer to the positive cluster or the negative cluster? "It doesn't happen on one message because you just send one message, but it happens because of all of your days' interactions are leading towards one of these two," Koneru said. "Then we say, okay, maybe this user is somebody who we need to take a much closer look at, and then we go pull all of their other conversations, other friends, and the games that they played, and all of those things." Humans review risky interactions and flag to law enforcement accordingly.

Roblox Rolls Out Open-Source AI System to Protect Kids From Predators in Chats

Roblox, the online gaming platform wildly popular with children and teenagers, is rolling out an open-source version of an artificial intelligence system it says can help preemptively detect predatory language in game chats. The move comes as the company faces lawsuits and criticism accusing it of not doing enough to protect children from predators. For instance, a lawsuit filed last month in Iowa alleges that a 13-year-old girl was introduced to an adult predator on Roblox, then kidnapped and trafficked across multiple states and raped. The suit, filed in Iowa District Court in Polk County, claims that Roblox's design features make children who use it "easy prey for pedophiles." Roblox says it strives to make its systems as safe as possible by default but notes that "no system is perfect, and one of the biggest challenges in the industry is to detect critical harms like potential child endangerment." The AI system, called Sentinel, helps detect early signs of possible child endangerment, such as sexually exploitative language. Roblox says the system has led the company to submit 1,200 reports of potential attempts at child exploitation to the National Center for Missing and Exploited Children in the first half of 2025. The company is now in the process of open-sourcing it so other platforms can use it too. Preemptively detecting possible dangers to kids can be tricky for AI systems -- and humans, too -- because conversations can seem innocuous at first. Questions like "how old are you?" or "where are you from?" wouldn't necessarily raise red flags on their own, but when put in context over the course of a longer conversation, they can take on a different meaning. Roblox, which has more than 111 million monthly users, doesn't allow users to share videos or images in chats and tries to block any personal information such as phone numbers, though -- as with most moderation rules -- people constantly find ways to get around such safeguards. It also doesn't allow kids under 13 to chat with other users outside of games unless they have explicit parental permission -- and unlike many other platforms, it does not encrypt private chat conversations, so it can monitor and moderate them. "We've had filters in place all along, but those filters tend to focus on what is said in a single line of text or within just a few lines of text. And that's really good for doing things like blocking profanity and blocking different types of abusive language and things like that," said Matt Kaufman, chief safety officer at Roblox. "But when you're thinking about things related to child endangerment or grooming, the types of behaviors you're looking at manifest over a very long period of time." Sentinel captures one-minute snapshots of chats across Roblox -- about 6 billion messages per day -- and analyzes them for potential harms. To do this, Roblox says it developed two indexes -- one made up of benign messages and, the other, chats that were determined to contain child endangerment violations. Roblox says this lets the system recognize harmful patterns that go beyond simply flagging certain words or phrases, taking the entire conversation into context. "That index gets better as we detect more bad actors, we just continuously update that index. Then we have another sample of what does a normal, regular user do?" said Naren Koneru, vice president of engineering for trust and safety at Roblox. As users are chatting, the system keeps score -- are they closer to the positive cluster or the negative cluster? "It doesn't happen on one message because you just send one message, but it happens because of all of your days' interactions are leading towards one of these two," Koneru said. "Then we say, okay, maybe this user is somebody who we need to take a much closer look at, and then we go pull all of their other conversations, other friends, and the games that they played, and all of those things." Humans review risky interactions and flag to law enforcement accordingly.

Roblox rolls out open-source AI system to protect kids from predators in chats - The Economic Times

Roblox is releasing an open-source AI system, Sentinel, designed to proactively identify predatory language in game chats, addressing criticisms and lawsuits concerning child safety on the platform. Sentinel analyzes chat snapshots, comparing them to indexes of benign and harmful conversations to detect potential child endangerment.Roblox, the online gaming platform wildly popular with children and teenagers, is rolling out an open-source version of an artificial intelligence system it says can help preemptively detect predatory language in game chats. The move comes as the company faces lawsuits and criticism accusing it of not doing enough to protect children from predators. For instance, a lawsuit filed last month in Iowa alleges that a 13-year-old girl was introduced to an adult predator on Roblox, then kidnapped and trafficked across multiple states and raped. The suit, filed in Iowa District Court in Polk County, claims that Roblox's design features make children who use it "easy prey for pedophiles." Roblox says it strives to make its systems as safe as possible by default but notes that "no system is perfect, and one of the biggest challenges in the industry is to detect critical harms like potential child endangerment." The AI system, called Sentinel, helps detect early signs of possible child endangerment, such as sexually exploitative language. Roblox says the system has led the company to submit 1,200 reports of potential attempts at child exploitation to the National Center for Missing and Exploited Children in the first half of 2025. The company is now in the process of open-sourcing it so other platforms can use it too. Preemptively detecting possible dangers to kids can be tricky for AI systems - and humans, too - because conversations can seem innocuous at first. Questions like "how old are you?" or "where are you from?" wouldn't necessarily raise red flags on their own, but when put in context over the course of a longer conversation, they can take on a different meaning. Roblox, which has more than 111 million monthly users, doesn't allow users to share videos or images in chats and tries to block any personal information such as phone numbers, though - as with most moderation rules - people constantly find ways to get around such safeguards. It also doesn't allow kids under 13 to chat with other users outside of games unless they have explicit parental permission - and unlike many other platforms, it does not encrypt private chat conversations, so it can monitor and moderate them. "We've had filters in place all along, but those filters tend to focus on what is said in a single line of text or within just a few lines of text. And that's really good for doing things like blocking profanity and blocking different types of abusive language and things like that," said Matt Kaufman, chief safety officer at Roblox. "But when you're thinking about things related to child endangerment or grooming, the types of behaviors you're looking at manifest over a very long period of time." Sentinel captures one-minute snapshots of chats across Roblox - about 6 billion messages per day - and analyzes them for potential harms. To do this, Roblox says it developed two indexes - one made up of benign messages and, the other, chats that were determined to contain child endangerment violations. Roblox says this lets the system recognize harmful patterns that go beyond simply flagging certain words or phrases, taking the entire conversation into context. "That index gets better as we detect more bad actors, we just continuously update that index. Then we have another sample of what does a normal, regular user do?" said Naren Koneru, vice president of engineering for trust and safety at Roblox. As users are chatting, the system keeps score - are they closer to the positive cluster or the negative cluster? "It doesn't happen on one message because you just send one message, but it happens because of all of your days' interactions are leading towards one of these two," Koneru said. "Then we say, okay, maybe this user is somebody who we need to take a much closer look at, and then we go pull all of their other conversations, other friends, and the games that they played, and all of those things." Humans review risky interactions and flag to law enforcement accordingly.

Roblox rolls out open-source AI system to protect kids from predators in chats

Roblox, the online gaming platform wildly popular with children and teenagers, is rolling out an open-source version of an artificial intelligence system it says can help preemptively detect predatory language in game chats. The move comes as the company faces lawsuits and criticism accusing it of not doing enough to protect children from predators. For instance, a lawsuit filed last month in Iowa alleges that a 13-year-old girl was introduced to an adult predator on Roblox, then kidnapped and trafficked across multiple states and raped. The suit, filed in Iowa District Court in Polk County, claims that Roblox's design features make children who use it "easy prey for pedophiles." Roblox says it strives to make its systems as safe as possible by default but notes that "no system is perfect, and one of the biggest challenges in the industry is to detect critical harms like potential child endangerment." The AI system, called Sentinel, helps detect early signs of possible child endangerment, such as sexually exploitative language. Roblox says the system has led the company to submit 1,200 reports of potential attempts at child exploitation to the National Center for Missing and Exploited Children in the first half of 2025. The company is now in the process of open-sourcing it so other platforms can use it too. Preemptively detecting possible dangers to kids can be tricky for AI systems -- and humans, too -- because conversations can seem innocuous at first. Questions like "how old are you?" or "where are you from?" wouldn't necessarily raise red flags on their own, but when put in context over the course of a longer conversation, they can take on a different meaning. Roblox, which has more than 111 million monthly users, doesn't allow users to share videos or images in chats and tries to block any personal information such as phone numbers, though -- as with most moderation rules -- people constantly find ways to get around such safeguards. It also doesn't allow kids under 13 to chat with other users outside of games unless they have explicit parental permission -- and unlike many other platforms, it does not encrypt private chat conversations, so it can monitor and moderate them. "We've had filters in place all along, but those filters tend to focus on what is said in a single line of text or within just a few lines of text. And that's really good for doing things like blocking profanity and blocking different types of abusive language and things like that," said Matt Kaufman, chief safety officer at Roblox. "But when you're thinking about things related to child endangerment or grooming, the types of behaviors you're looking at manifest over a very long period of time." Sentinel captures one-minute snapshots of chats across Roblox -- about 6 billion messages per day -- and analyzes them for potential harms. To do this, Roblox says it developed two indexes -- one made up of benign messages and, the other, chats that were determined to contain child endangerment violations. Roblox says this lets the system recognize harmful patterns that go beyond simply flagging certain words or phrases, taking the entire conversation into context. "That index gets better as we detect more bad actors, we just continuously update that index. Then we have another sample of what does a normal, regular user do?" said Naren Koneru, vice president of engineering for trust and safety at Roblox. As users are chatting, the system keeps score -- are they closer to the positive cluster or the negative cluster? "It doesn't happen on one message because you just send one message, but it happens because of all of your days' interactions are leading towards one of these two," Koneru said. "Then we say, okay, maybe this user is somebody who we need to take a much closer look at, and then we go pull all of their other conversations, other friends, and the games that they played, and all of those things." Humans review risky interactions and flag to law enforcement accordingly.

Roblox Releases Early Warning System to Help Keep Children Safer Online

Roblox Corporation announced Roblox Sentinel, an artificial intelligence (AI) system that helps detect child endangerment communications early so Roblox can investigate instances sooner and flag to law enforcement when needed. Sentinel is open source and available for other digital platforms across the industry to utilize. As part of Roblox's open-source safety toolkit, Sentinel is an early- warning system designed to identify subtle, hard-to-detect communication patterns that could indicate potential grooming. Since late 2024, Sentinel has been helping Roblox take prompt action and investigate to help keep users safer. In the first half of 2025, Sentinel aided team in submitting approximately 1,200 reports of potential child exploitation attempts to the National Center for missing and Exploited Children. A strategic blend of experienced humans and AI is imperative in scaling safety on Roblox, including detecting potential signals identified by Sentinel. Expert analysts review cases that the AI system flags as potentially violations. The analysts' decisions create a cycle that allows Roblox to continuously tailor and refine the samples, indexes, and training sets. This process with humans is vital in helping Sentinel adjust and stay up to date with new and developing patterns of bad actors. Roblox works to improve safety across the industry wherever possible by open-sourcing innovative technical solutions, as well as collaborating with partners including ROOST, in which Roblox is a founding partner, and the Tech Coalition's Lantern project.