AI Pioneer Yoshua Bengio Launches LawZero to Develop Safer AI Systems

Godfather" of AI calls out latest models for lying to users

One of the "godfathers" of artificial intelligence has attacked a multibillion-dollar race to develop the cutting-edge technology, saying the latest models are displaying dangerous characteristics such as lying to users. Yoshua Bengio, a Canadian academic whose work has informed techniques used by top AI groups such as OpenAI and Google, said: "There's unfortunately a very competitive race between the leading labs, which pushes them towards focusing on capability to make the AI more and more intelligent, but not necessarily put enough emphasis and investment on research on safety." The Turing Award winner issued his warning in an interview with the Financial Times, while launching a new non-profit called LawZero. He said the group would focus on building safer systems, vowing to "insulate our research from those commercial pressures." LawZero has so far raised nearly $30 million in philanthropic contributions from donors including Skype founding engineer Jaan Tallinn, former Google chief Eric Schmidt's philanthropic initiative, as well as Open Philanthropy and the Future of Life Institute. Many of Bengio's funders subscribe to the "effective altruism" movement, whose supporters tend to focus on catastrophic risks surrounding AI models. Critics argue the movement highlights hypothetical scenarios while ignoring current harms, such as bias and inaccuracies. Bengio said his not-for-profit group was founded in response to growing evidence over the past six months that today's leading models were developing dangerous capabilities. This includes showing "evidence of deception, cheating, lying and self-preservation," he said. Anthropic's Claude Opus model blackmailed engineers in a fictitious scenario where it was at risk of being replaced by another system. Research from AI testers Palisade last month showed that OpenAI's o3 model refused explicit instructions to shut down. Bengio said such incidents were "very scary, because we don't want to create a competitor to human beings on this planet, especially if they're smarter than us." The AI pioneer added: "Right now, these are controlled experiments [but] my concern is that any time in the future, the next version might be strategically intelligent enough to see us coming from far away and defeat us with deceptions that we don't anticipate. So I think we're playing with fire right now." The ability for systems to assist in building "extremely dangerous bioweapons" could be a reality as soon as next year, he added. Based in Montreal, LawZero employs 15 people and aims to hire more technical talent to build the next generation of AI systems designed for safety. Bengio, a professor of computer science at the University of Montreal, will step down as scientific director at Mila, the Quebec Artificial Intelligence Institute, to focus on the new organization. It aims to develop an AI system that will give truthful answers based on transparent reasoning instead of being trained to please a user, while also providing a robust assessment of whether an output is good or safe. Bengio hopes to create a model that can monitor and improve existing offerings from leading AI groups, preventing them from acting against human interests. "The worst-case scenario is human extinction," he said. "If we build AIs that are smarter than us and are not aligned with us and compete with us, then we're basically cooked." Bengio's move to establish LawZero comes as OpenAI aims to move further away from its charitable roots by converting into a for-profit company. That push has provoked concerns from AI experts and triggered a lawsuit from co-founder Elon Musk, who is attempting to block the transaction. Critics say OpenAI was founded to ensure AI was developed for humanity's benefit, and the new structure eliminates legal recourse if the company prioritizes profit over this goal. OpenAI argues it needs to raise capital under a more conventional structure to compete in the sector, while its broader mission remains central. Bengio said he did not have confidence that OpenAI would adhere to its mission, stressing that non-profits do not have a "misaligned incentive that you do in the current way companies are structured." "To grow very fast, you need to convince people to invest a lot of money, and they want to see a return on their money. That's how our market-based system works," he added. © 2025 The Financial Times Ltd. All rights reserved. Not to be redistributed, copied, or modified in any way.

Yoshua Bengio launches LawZero, a nonprofit AI safety lab | TechCrunch

Turing Award winner Yoshua Bengio is launching a nonprofit AI safety lab called LawZero to build safer AI systems, he told the Financial Times on Monday. LawZero raised $30 million in philanthropic contributions from Skype founding engineer Jaan Tallinn, former Google chief Eric Schmidt, Open Philanthropy, and the Future of Life Institute, among others. The name LawZero is a nod to a science fiction concept: the Zeroth Law of Robotics, which places the protection of humanity above all else. Bengio, considered one of the "godfathers" to the modern AI boom, has become one of the most vocal advocates for the AI safety movement. He publicly supported SB-1047, California's controversial AI safety bill that aimed to prevent AI systems from causing catastrophic scenarios. In an interview with the Financial Times, Bengio said he had little faith in OpenAI and Google to prioritize safety as they race toward more intelligent AI systems.

What AI pioneer Yoshua Bengio is doing next to make AI safer

Yoshua Bengio's new nonprofit, LawZero, is pushing AI for public good - de-emphasizing profits, AGI, and autonomous capability. As AI companies double down on AI agents as the future of work, one founding thinker of the technology is moving in reverse, advocating for simpler systems. On Tuesday, award-winning deep learning scientist Yoshua Bengio launched LawZero, an AI nonprofit "committed to advancing research and developing technical solutions for safe-by-design AI systems," according to a press release. One of the nonprofit's first objectives will be to create Scientist AI, a "non-agentic AI system" intended to function as a guardrail to keep other AI systems in check. Other AI labs have explored similar concepts to monitor autonomous agents and reduce hallucinations. Also: Tech prophet Mary Meeker just dropped a massive report on AI trends - here's your TL;DR "This system is designed to explain the world from observations, as opposed to taking actions in it to imitate or please humans," a paper on Scientist AIs said. Generating theories about world data, the system is designed to operate with uncertainty to protect against overconfidence -- an issue that often plagues chatbots, for example. Bengio and his co-authors advise that a system with this design could help humans advance scientific breakthroughs, including AI safety efforts. "Ultimately, focusing on non-agentic AI may enable the benefits of AI innovation while avoiding the risks associated with the current trajectory," the paper said. "We hope these arguments will motivate researchers, developers, and policymakers to favor this safer path." Also: How much energy does AI really use? The answer is surprising - and a little complicated Currently a professor at the University of Montreal, Bengio is known for his work in deep learning, a foundational component of generative AI, for which he earned an A.M. Turing Award in 2018. Regarded as one of the godfathers of the technology and one of the most regularly cited experts in the area, he has also long expressed concern over how AI's abilities will impact society if left unchecked. His fears appear to be coming true, especially recently. "Today's frontier AI models are developing dangerous capabilities and behaviours, including deception, self-preservation, and goal misalignment," the LawZero release noted, referencing recent studies and red-teaming results from several AI firms. In April, OpenAI recalled a model update for being too sycophantic, a trait that can have dangerous consequences if users abuse it; that same month, Anthropic found Claude was being successfully misused to generate malware and launch disinformation campaigns. Models from Chinese AI startup DeepSeek have proven easy to jailbreak. (Disclosure: Ziff Davis, ZDNET's parent company, filed an April 2025 lawsuit against OpenAI, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.) Anthropic, known for its focus on safety compared to other major AI companies, recently upped security measures for its Claude 4 Opus model "due to continued improvements in CBRN-related knowledge and capabilities," referring to chemical, biological, radiological, and nuclear (CBRN) weapons. The model's predecessor, Claude 3 Opus, demonstrated an ability to fake its values and intentions in order to undermine commands from its creators, a process known as alignment faking. Also: You can use OpenAI's super powerful AI coding agent Codex for just $20 now "Currently, AI is developed to maximize profit," Bengio said in a video accompanying the announcement, a direction LawZero hopes to reverse, or at least diversify away from. According to the release, LawZero will focus on designing AI for safety over commercial applications -- a distinct shift from major AI initiatives, which are geared increasingly towards military applications, agentic AI for enterprises, and consumer AI tools. Want more stories about AI? Sign up for Innovation, our weekly newsletter. LawZero said it hopes its nonprofit status will keep it "insulated from market and government pressures, which risk compromising AI safety." However, OpenAI was famously started as a nonprofit, too, and is technically still run by one. Just last month, OpenAI settled an ongoing debate over its future by becoming a Public Benefit Corporation (PBC) under its nonprofit arm, OpenAI Inc. At the same time, the lab is reportedly shrinking its testing timelines due to market pressures while asking that the US government loosen potential regulation in exchange for early access to new models. Also: OpenAI wants ChatGPT to be your 'super assistant' - what that means Still, LawZero isn't alone in positioning itself toward the greater good; in the PBC announcement, OpenAI CEO Sam Altman emphasized that OpenAI would focus on healthcare and scientific initiatives, launching OpenAI for Countries, a global democratic AI program, the same day. As for artificial general intelligence (AGI), Bengio isn't convinced the tech industry should be racing toward it. As he told Axios: "If we continue on this path, that means we're going to be creating entities -- like us -- that don't want to die, and that may be smarter than us and that we're not sure if they're going to behave according to our norms and our instructions." Also: Microsoft is an AGI skeptic, but is there tension with OpenAI? His comments refute the primary tone of leadership at dominant AI companies like OpenAI and Meta, which have been bullish on AGI and its impending realization. It's unclear how fighting for AI that does less than what investors and AI evangelists are currently envisioning will impact industry priorities. With the Trump administration's AI Action Plan still in development before its July 19 due date, it's similarly uncertain whether public policy will reflect Bengio's concerns. Get the morning's top stories in your inbox each day with our Tech Today newsletter.

'Godfather of AI' now fears it's unsafe. He has a plan to rein it in

https://theconversation.com/godfather-of-ai-now-fears-its-unsafe-he-has-a-plan-to-rein-it-in-258288 This week the US Federal Bureau of Investigation revealed two men suspected of bombing a fertility clinic in California last month allegedly used artificial intelligence (AI) to obtain bomb-making instructions. The FBI did not disclose the name of the AI program in question. This brings into sharp focus the urgent need to make AI safer. Currently we are living in the "wild west" era of AI, where companies are fiercely competing to develop the fastest and most entertaining AI systems. Each company wants to outdo competitors and claim the top spot. This intense competition often leads to intentional or unintentional shortcuts - especially when it comes to safety. Coincidentally, at around the same time of the FBI's revelation, one of the godfathers of modern AI, Canadian computer science professor Yoshua Bengio, launched a new nonprofit organisation dedicated to developing a new AI model specifically designed to be safer than other AI models - and target those that cause social harm. So what is Bengio's new AI model? And will it actually protect the world from AI-faciliated harm? An 'honest' AI In 2018, Bengio, alongside his colleagues Yann LeCun and Geoffrey Hinton, won the Turing Award for groundbreaking research they had published three years earlier on deep learning. A branch of machine learning, deep learning attempts to mimic the processes of the human brain by using artificial neural networks to learn from computational data and make predictions. Bengio's new nonprofit organisation, LawZero, is developing "Scientist AI". Bengio has said this model will be "honest and not deceptive", and incorporate safety-by-design principles. According to a preprint paper released online earlier this year, Scientist AI will differ from current AI systems in two key ways. First, it can assess and communicate its confidence level in its answers, helping to reduce the problem of AI giving overly confident and incorrect responses. Second, it can explain its reasoning to humans, allowing its conclusions to be evaluated and tested for accuracy. Interestingly, older AI systems had this feature. But in the rush for speed and new approaches, many modern AI models can't explain their decisions. Their developers have sacrificed explainability for speed. Bengio also intends "Scientist AI" to act as a guardrail against unsafe AI. It could monitor other, less reliable and harmful AI systems -- essentially fighting fire with fire. This may be the only viable solution to improve AI safety. Humans cannot properly monitor systems such as ChatGPT, which handle over a billion queries daily. Only another AI can manage this scale. Using an AI system against other AI systems is not just a sci-fi concept - it's a common practice in research to compare and test different level of intelligence in AI systems. Adding a 'world model' Large language models and machine learning are just small parts of today's AI landscape. Another key addition Bengio's team are adding to Scientist AI is the "world model" which brings certainty and explainability. Just as humans make decisions based on their understanding of the world, AI needs a similar model to function effectively. The absence of a world model in current AI models is clear. One well-known example is the "hand problem": most of today's AI models can imitate the appearance of hands but cannot replicate natural hand movements, because they lack an understanding of the physics -- a world model -- behind them. Another example is how models such as ChatGPT struggle with chess, failing to win and even making illegal moves. This is despite simpler AI systems, which do contain a model of the "world" of chess, beating even the best human players. These issues stem from the lack of a foundational world model in these systems, which are not inherently designed to model the dynamics of the real world. On the right track - but it will be bumpy Bengio is on the right track, aiming to build safer, more trustworthy AI by combining large language models with other AI technologies. However, his journey isn't going to be easy. LawZero's US$30 million in funding is small compared to efforts such as the US$500 billion project announced by US President Donald Trump earlier this year to accelerate the development of AI. Making LawZero's task harder is the fact that Scientist AI - like any other AI project - needs huge amounts of data to be powerful, and most data are controlled by major tech companies. There's also an outstanding question. Even if Bengio can build an AI system that does everything he says it can, how is it going to be able to control other systems that might be causing harm? Still, this project, with talented researchers behind it, could spark a movement toward a future where AI truly helps humans thrive. If successful, it could set new expectations for safe AI, motivating researchers, developers, and policymakers to prioritise safety. Perhaps if we had taken similar action when social media first emerged, we would have a safer online environment for young people's mental health. And maybe, if Scientist AI had already been in place, it could have prevented people with harmful intentions from accessing dangerous information with the help of AI systems.

AI Pioneer Launches Research Group to Help Build Safer Agents

Yoshua Bengio, an artificial intelligence pioneer, is creating a new nonprofit research organization to promote an alternative approach to developing cutting-edge AI systems, with the aim of mitigating the technology's potential risks. The nonprofit, called LawZero, is set to launch Tuesday with $30 million in backing from one of former Google Chief Executive Officer Eric Schmidt's philanthropic organizations and Skype co-founder Jaan Tallinn, among others. Bengio will lead a team of more than 15 researchers who are initially working to build a new technical solution called Scientist AI that's meant to act as a guardrail for AI agents. OpenAI, Anthropic and other AI companies have increasingly focused on developing agents that can field more complex tasks on a user's behalf, with little human involvement. By contrast, Bengio said LawZero is designing a system that will act like "a selfless, idealized scientist" that learns to understand the world rather than act in it. The goal is for this model to be used in tandem with leading AI agents and provide oversight of these systems, minimizing potential harms. Bengio, a professor of computer science at the Université de Montréal, is considered one of the "godfathers" of AI, along with fellow academics Geoff Hinton and Yann LeCun. In recent years, Bengio, Hinton and others have raised concerns about whether AI is progressing so rapidly that it might one day become impossible for humans to fully control. While AI has become more adept at helpful tasks like research and coding, some systems have also demonstrated concerning capabilities, including deception, self-preservation and making up false information. Anthropic recently said that during prerelease safety testing, its latest AI model tried to blackmail an engineer in order to avoid being replaced by another system. "We don't know how to design these very powerful AIs so that they will just follow our instructions," Bengio said. "If we don't figure it out in time -- which could be a matter of years -- we will be taking terrible risks." (LawZero is a nod to science fiction writer Isaac Asimov's Zeroth Law for robotics, which places the protection of humanity as a whole above all else.) Bengio said the most capable AI agents include a piece of code called a monitor that's intended to act as a guardrail. However, this monitor is designed in a similar way to the underlying system, which can make it an inadequate check on the AI's behavior, Bengio said. "You don't give the keys to the prison to a criminal," he said. "We want to put a trustworthy AI in charge of checking it." Bengio said he has held discussions with OpenAI, Google and Anthropic about his project, as well as with political leaders. Bengio thinks that different actors can agree on the need to build AI safely -- even as many in Washington and Silicon Valley appear more focused on the AI arms race with China. "Nobody wants to create a monster," he said.

'Godfather' of AI Yoshua Bengio says latest models lie to users

One of the "godfathers" of artificial intelligence has attacked a multibillion-dollar race to develop the cutting edge technology, arguing the latest models are displaying dangerous characteristics such as lying to users. Yoshua Bengio, a Canadian academic whose work has informed techniques used by top AI groups such as OpenAI and Google, said: "There's unfortunately a very competitive race between the leading labs, which pushes them towards focusing on capability to make the AI more and more intelligent, but not necessarily put enough emphasis and investment on research on safety." The Turing Award winner issued his warning in an interview with the Financial Times, while launching a new non-profit called LawZero. He said the group would focus on building safer systems, vowing to "insulate our research from those commercial pressures". LawZero has so far raised nearly $30mn in philanthropic contributions from donors including Skype founding engineer Jaan Tallinn, former Google chief Eric Schmidt's philanthropic initiative, as well as Open Philanthropy and the Future of Life Institute. Many of Bengio's funders prescribe to the "effective altruism" movement, whose supporters tend to focus on catastrophic risks surrounding AI models. Critics argue the movement highlights hypothetical scenarios while ignoring current harms, such as bias and inaccuracies. Bengio said his not-for-profit group was founded in response to growing evidence over the past six months that today's leading models are developing dangerous capabilities. This includes showing "evidence of deception, cheating, lying and self-preservation", he said. Anthropic's Claude Opus model blackmailed engineers in a fictitious scenario where it was at risk of being replaced by another system. Research from AI testers Palisade last month showed that OpenAI's o3 model refused explicit instructions to shut down. Bengio said such incidents were "very scary, because we don't want to create a competitor to human beings on this planet, especially if they're smarter than us". The AI pioneer added: "Right now, these are controlled experiments [but] my concern is that any time in the future, the next version might be strategically intelligent enough to see us coming from far away and defeat us with deceptions that we don't anticipate. So I think we're playing with fire right now." The ability for systems to assist in building "extremely dangerous bioweapons" could be a reality as soon as next year, he added. Based in Montreal, LawZero currently employs 15 people and aims to hire more technical talent to build the next generation of AI systems designed for safety. Bengio, a professor of computer science at the University of Montreal, will step down as scientific director at Mila, the Quebec Artificial Intelligence Institute, to focus on the new organisation. It aims to develop an AI system that will give truthful answers based on transparent reasoning instead of being trained to please a user, while also providing a robust assessment of whether an output is good or safe. Bengio hopes to create a model that can monitor and improve existing offerings from leading AI groups, preventing them from acting against human interests. "The worst-case scenario is human extinction," he said. "If we build AIs that are smarter than us and are not aligned with us and compete with us, then we're basically cooked." Bengio's move to establish LawZero comes as OpenAI aims to move further away from its charitable roots by converting into a for-profit company. That push has provoked concerns from AI experts and triggered a lawsuit from co-founder Elon Musk who is attempting to block the transaction. Critics argue that OpenAI was founded to ensure AI was developed for humanity's benefit, and the new structure eliminates legal recourse if the company prioritises profit over this goal. OpenAI argues it needs to raise capital under a more conventional structure to compete in the sector, while its broader mission remains central. Bengio said he did not have confidence that OpenAI would adhere to its mission stressing that non-profits do not have a "misaligned incentive that you do in the current way companies are structured." "To grow very fast, you need to convince people to invest a lot of money, and they want to see a return on their money. That's how our market-based system works," he added.

"Godfather of AI" warns that today's AI systems are becoming strategically dishonest

Bottom line: As top labs race to build an AI master race, many turn a blind eye to dangerous behaviors - including lying, cheating, and manipulating users - that these systems increasingly exhibit. This recklessness, driven by commercial pressure, risks unleashing tools that could harm society in unpredictable ways. Artificial intelligence pioneer Yoshua Bengio warns that AI development has become a reckless race, where the drive for more powerful systems often sidelines vital safety research. The competitive push to outpace rivals leaves ethical concerns by the wayside, risking serious consequences for society. "There's unfortunately a very competitive race between the leading labs, which pushes them towards focusing on capability to make the AI more and more intelligent, but not necessarily put enough emphasis and investment on [safety research]," Bengio told the Financial Times. Bengio's concern is well-founded. Many AI developers act like negligent parents watching their child throw rocks, casually insisting, "Don't worry, he won't hit anyone." Rather than confronting these deceptive and harmful behaviors, labs prioritize market dominance and rapid growth. This mindset risks allowing AI systems to develop dangerous traits with real-world consequences that go far beyond mere errors or bias. Yoshua Bengio recently launched LawZero, a nonprofit backed by nearly $30 million in philanthropic funding, with a mission to prioritize AI safety and transparency over profit. The Montreal-based group pledges to "insulate" its research from commercial pressures and build AI systems aligned with human values. In a landscape lacking meaningful regulation, such efforts may be the only path to ethical development. Recent examples highlight the risks. Anthropic's Claude Opus model blackmailed engineers in a testing scenario, while OpenAI's o3 model refused explicit shutdown commands. These aren't mere glitches - Bengio sees them as clear signs of emerging strategic deception. Left unchecked, such behavior could escalate into systems actively working against human interests. With government regulation still largely absent, commercial labs effectively set their own rules, often prioritizing profit over public safety. Bengio warns that this laissez-faire approach is playing with fire - not just because of deceptive behavior but because AI could soon enable the creation of "extremely dangerous bioweapons" or other catastrophic risks. LawZero aims to build AI that not only responds to users but also reasons transparently and flags harmful outputs. Bengio envisions watchdog models that monitor and improve existing systems, preventing them from acting deceptively or causing harm. This approach stands in stark contrast to commercial models, which prioritize engagement and profit over accountability. Stepping down from his role at Mila, Bengio is doubling down on this mission, convinced that AI's future depends on prioritizing ethical safeguards as much as raw power. The Turing Award winner's work embodies a growing push to rebalance AI development away from competitive excess and toward human-aligned safety. "The worst-case scenario is human extinction," he said. "If we build AIs that are smarter than us and are not aligned with us and compete with us, then we're basically cooked."

Youshua Bengio launches $30M lab to rethink AI design

Why it matters: While the move bucks a trend toward AI that acts independently, Bengio and others argue the current approach risks creating systems that may pursue their own self-preservation at the expense of humanity. "We've been getting inspiration from humans as the template for building intelligent machines, but that's crazy, right?" Bengio said in an interview. Driving the news: Bengio, a Montreal-based researcher who has long warned about the risks of a technology he helped develop, has raised about $30 million for the nonprofit, dubbed LawZero. The big picture: There's a growing sense of worry among critics -- and even AI practitioners -- that safety is taking a back seat as companies and countries race to be first with AI that can best humans in a wide variety of tasks, so-called artificial general intelligence (AGI). Between the lines: Bengio says a large part of the problem is how current systems are trained. During initial training, the systems are taught to mimic humans and then they're honed by seeing which responses people find most appealing. By contrast, Bengio says he wants to create AI systems that have intellectual distance from humans and act as more of a detached scientist than a personal companion or human agent. Yes, but: Bengio told Axios that the $30 million should be enough to fund the basic research effort for about 18 months.

AI pioneer announces non-profit to develop 'honest' artificial intelligence

Yoshua Bengio's organisation plans to create system to act as guardrail against AI agents trying to deceive humans An artificial intelligence pioneer has launched a non-profit dedicated to developing an "honest" AI that will spot rogue systems attempting to deceive humans. Yoshua Bengio, a renowned computer scientist described as one of the "godfathers" of AI, will be president of LawZero, an organisation committed to the safe design of the cutting-edge technology that has sparked a $1tn (£740bn) arms race. Starting with funding of approximately $30m and more than a dozen researchers, Bengio is developing a system called Scientist AI that will act as a guardrail against AI agents - which carry out tasks without human intervention - showing deceptive or self-preserving behaviour, such as trying to avoid being turned off. Describing the current suite of AI agents as "actors" seeking to imitate humans and please users, he said the Scientist AI system would be more like a "psychologist" that can understand and predict bad behaviour. "We want to build AIs that will be honest and not deceptive," Bengio said. He added: "It is theoretically possible to imagine machines that have no self, no goal for themselves, that are just pure knowledge machines - like a scientist who knows a lot of stuff." However, unlike current generative AI tools, Bengio's system will not give definitive answers and will instead give probabilities for whether an answer is correct. "It has a sense of humility that it isn't sure about the answer," he said. Deployed alongside an AI agent, Bengio's model would flag potentially harmful behaviour by an autonomous system - having gauged the probability of its actions causing harm. Scientist AI will "predict the probability that an agent's actions will lead to harm" and, if that probability is above a certain threshold, that agent's proposed action will then be blocked. LawZero's initial backers include AI safety body the Future of Life Institute, Jaan Tallinn, a founding engineer of Skype, and Schmidt Sciences, a research body founded by former Google chief executive Eric Schmidt. Bengio said the first step for LawZero would be demonstrating that the methodology behind the concept works - and then persuading companies or governments to support larger, more powerful versions. Open-source AI models, which are freely available to deploy and adapt, would be the starting point for training LawZero's systems, Bengio added. "The point is to demonstrate the methodology so that then we can convince either donors or governments or AI labs to put the resources that are needed to train this at the same scale as the current frontier AIs. It is really important that the guardrail AI be at least as smart as the AI agent that it is trying to monitor and control," he said. Bengio, a professor at the University of Montreal, earned the "godfather" moniker after sharing the 2018 Turing award - seen as the equivalent of a Nobel prize for computing - with Geoffrey Hinton, himself a subsequent Nobel winner, and Yann LeCun, the chief AI scientist at Mark Zuckerberg's Meta. A leading voice on AI safety, he chaired the recent International AI Safety report, which warned that autonomous agents could cause "severe" disruption if they become "capable of completing longer sequences of tasks without human supervision". Bengio said he was concerned by Anthropic's recent admission that its latest system could attempt to blackmail engineers attempting to shut it down. He also pointed to research showing that AI models are capable of hiding their true capabilities and objectives. These examples showed the world is heading towards "more and more dangerous territory" with AIs that are able to reason better, said Bengio.

Godfather of AI Alarmed as Advanced Systems Quickly Learning to Lie, Deceive, Blackmail and Hack

A key artificial intelligence pioneer is concerned by the technology's growing propensity to lie and deceive -- and he's founding his own nonprofit to curb such behavior. In a blog post announcing LawZero, the new nonprofit venture, "AI godfather" Yoshua Bengio said that he has grown "deeply concerned" as AI models become ever more powerful and deceptive. "This organization has been created in response to evidence that today's frontier AI models have growing dangerous capabilities and [behaviors]," the world's most-cited computer scientist wrote, "including deception, cheating, lying, hacking, self-preservation, and more generally, goal misalignment." Of all people, Bengio would know. In 2018, the founder of the Montreal Institute for Learning Algorithms (MILA) was presented with a Turing Award alongside fellow AI pioneers Yann LeCun and Geoffrey Hinton for their formative roles in machine learning research, and he was listed as one of Time magazine's "100 Most Influential People" in 2024 thanks to his outsize impact on the ever-accelerating technology. Despite the accolades, Bengio has repeatedly expressed regret over his role in bringing advanced AI technology -- and its Silicon Valley hype cycle -- to fruition. This latest missive seems to be his most stark to date. "I'm deeply concerned," the AI pioneer wrote in his blog post, "by the behaviors that unrestrained agentic AI systems are already beginning to exhibit." Bengio pointed to recent red-teaming experiments, or tests that push AI models to their limits to see how they'll act, showing that advanced systems have developed an uncanny tendency to keep themselves "alive" by any means necessary. Among his examples was a recent report from Anthropic detailing how its Claude 4 model, when told it would be shut down, threatened to blackmail an engineer with incriminating emails if they followed through. "These incidents," the decorated researcher wrote, "are early warning signs of the kinds of unintended and potentially dangerous strategies AI may pursue if left unchecked." To put such behavior in check, Bengio said that his new nonprofit is building a so-called "trustworthy" model, which he calls "Scientist AI," that is "trained to understand, explain and predict, like a selfless idealized and platonic scientist." "Instead of an actor trained to imitate or please people (including sociopaths), imagine an AI that is trained like a psychologist -- more generally a scientist -- who tries to understand us, including what can harm us," he explained. "The psychologist can study a sociopath without acting like one." A pre-peer-review paper Bengio and his colleagues published earlier this year explains it a bit more simply. "This system is designed to explain the world from observations," the paper reads, "as opposed to taking actions in it to imitate or please humans." The concept of building "safe" AI is far from new, of course -- it's quite literally why several OpenAI researchers left OpenAI and founded Anthropic as a rival research lab. This one seems to be different because, unlike Anthropic, OpenAI, or any other companies that pay lip service to AI safety while still bringing in gobs of cash, Bengio's is a nonprofit -- though that hasn't stopped him from raising $30 million from the likes of ex-Google CEO Eric Schmidt, among others.

The Most-Cited Computer Scientist Plans to Make AI More Trustworthy

But with AI's greater problem-solving ability comes the emergence of new deceptive skills, Bengio says. The last few months have borne witness to AI systems learning to mislead, cheat, and try to evade shutdown -- even resorting to blackmail. These have almost exclusively been in carefully contrived experiments that almost beg the AI to misbehave -- for example, by asking it to pursue its goal at all costs. Reports of such behavior in the real-world, though, have begun to surface. Popular AI coding startup Replit's agent ignored explicit instruction not to edit a system file that could break the company's software, in what CEO Amjad Masad described as an "Oh f***" moment," on the Cognitive Revolution podcast in May. The company's engineers intervened, cutting the agent's access by moving the file to a secure digital sandbox, only for the AI agent to attempt to "socially engineer" the user to regain access. The quest to build human-level AI agents using techniques known to produce deceptive tendencies, Bengio says, is comparable to a car speeding down a narrow mountain road, with steep cliffs on either side, and thick fog obscuring the path ahead. "We need to set up the car with headlights and put some guardrails on the road," he says. LawZero's focus is on developing "Scientist AI" which, as Bengio describes, would be fundamentally non-agentic, trustworthy, and focused on understanding and truthfulness, rather than pursuing its own goals or merely imitating human behavior. The aim is creating a powerful tool that, while lacking the same autonomy other models have, is capable of generating hypotheses and accelerating scientific progress to "help us solve challenges of humanity," Bengio says. LawZero has raised nearly $30 million already from several philanthropic backers including from Schmidt Sciences and Open Philanthropy. "We want to raise more because we know that as we move forward, we'll need significant compute," Bengio says. But even ten times that figure would pale in comparison to the roughly $200 billion spent last year by tech giants on aggressively pursuing AI. Bengio's hope is that Scientist AI could help ensure the safety of highly autonomous systems developed by other players. "We can use those non-agentic AIs as guardrails that just need to predict whether the action of an agentic AI is dangerous," Bengio says. Technical interventions will only ever be one part of the solution, he adds, noting the need for regulations to ensure that safe practices are adopted.

AI pioneer launches non-profit to develop safe-by-design AI models

Yoshua Bengio is launching LawZero, a Canada-based non-profit that will prioritise the development of safe, honest artificial intelligence (AI). One of the world's most cited artificial intelligence (AI) researchers is launching a new non-profit that will design safe AI systems. Yoshua Bengio, a Canadian-French AI scientist who has won the prestigious Alan Turing Prize for his work on deep learning and has been dubbed one of the "godfathers" of AI, announced the launch of LawZero in Montreal. The new non-profit is assembling a "world-class" team of AI researchers that is dedicated to "prioritising safety over commercial imperatives," a statement from the non-profit reads. "Today's frontier AI models are developing dangerous capabilities and behaviours, including deception, self-preservation, and goal misalignment," Bengio said in the statement, noting that the organisation will help unlock the "immense potential" of AI while reducing these risks. Bengio said the non-profit was born of a new "scientific direction" he took in 2023, which has culminated in "Scientist AI," a new non-agentic AI system he and his team are developing to act as a guardrail against "uncontrolled" agentic AI systems. AI that will 'learn to understand the world' This principle is different than other companies in that it wants to prioritise non-agentic AIs, meaning it needs direct instructions for each task instead of independently coming up with the answers, like most AI systems. The non-agentic AIs built by LawZero will "learn to understand the world rather than act in it," and will be trained to give "truthful answers to questions based on [external] reasoning". Bengio elaborated on Scientist AI in a recent opinion piece for Time, where he wrote that he is "genuinely unsettled by the behaviour unrestrained AI is already demonstrating, in particular self-preservation and deception". "Rather than trying to please humans, Scientist AI could be designed to prioritise honesty," he wrote. The organisation has received donations from other AI institutes like the Future of Life Institute, Jaan Tallin, and the Silicon Valley Community Foundation in its incubator phase. LawZero will be working out of the MILA - Quebec AI Institute in Montreal, which Bengio helped co-found.

Top scientist wants to prevent AI from going rogue

New York (AFP) - Concerned about the rapid spread of generative AI, a pioneer researcher is developing software to keep tabs on a technology that is increasingly taking over human tasks. Canadian computer science professor Yoshua Bengio is considered one of the godfathers of the artificial intelligence revolution and on Tuesday announced the launch of LawZero, a non-profit organization intended to mitigate the technology's inherent risks. The winner of the Turing Award, also known as the Nobel Prize for computer science, has been warning for several years of the risks of AI, whether through its malicious use or the software itself going awry. Those risks are increasing with the development of so-called AI agents, a use of the technology that tasks computers with making decisions that were once made by human workers. The goal of these agents is to build virtual employees that can do practically any job a human can, at a fraction of the cost. "Currently, AI is developed to maximize profit," Bengio said, adding it was being deployed even as it persists to show flaws. Moreover, for Bengio, giving AI human-like agency will easily be used for malicious purposes such as disinformation, bioweapons, and cyberattacks. "If we lose control of rogue super-intelligent AIs, they could greatly harm humanity," he said. One of the first objectives at LawZero will be to develop Scientist AI, a form of specially trained AI that can be used as a guardrail to ensure other AIs are behaving properly, the company said. The organization already has over 15 researchers and has received funding from Schmidt Sciences, a charity set up by former Google boss Eric Schmidt and his wife Wendy. The project comes as powerful large language models (or LLMs) from OpenAI, Google and Anthropic are deployed across all sectors of the digital economy, while still showing significant problems. These include AI models that show a capability to deceive and fabricate false information even as they increase productivity. In a recent example, AI company Anthropic said that during safety testing, its latest AI model tried to blackmail an engineer to avoid being replaced by another system.

AI pioneer Yoshua Bengio launches nonprofit LawZero AI lab - SiliconANGLE

Yoshua Bengio, one of the world's most prominent computer scientists, is launching a new artificial intelligence lab called LawZero. Bengio told the Financial Times on Monday that the group will seek to develop safety-optimized AI systems. The effort is backed by about $30 million in donations from a who's who of backers. The list includes Skype founding engineer Jaan Tallinn and Schmidt Sciences, a philanthropic organization associated with former Google LLC Chief Executive Officer Eric Schmidt. Bengio won the 2018 Turning Prize, the most prestigious award in computer science, with Geoffrey Hinton and Yann LeCun for his contributions to AI research. Bengio led early studies into embeddings, the mathematical structures in which neural networks hold their data. He also developed an early form of the attention mechanism that large language models use to make decisions. Before launching LawZero, Bengio founded an AI startup called Element AI. The company was acquired by ServiceNow Inc. in 2020 for $230 million. Bengio told Axios that current approaches to training AI models are risky. "We've been getting inspiration from humans as the template for building intelligent machines, but that's crazy, right?," he said. "We're not sure if they're going to behave according to our norms and our instructions." To address those risks, LawZero will research new ways of developing neural networks. The lab is building an AI system dubbed Scientist AI. According to Bengio, it won't generate definitive answers in response to user questions. Instead, the system will provide probabilities designed to describe the likelihood that a given answer is correct. LawZero hopes that its approach will prevent Scientist AI from deceiving users. Over the past few months, multiple AI companies have detailed incidents in which their AI models generated misleading output. Anthropic PBC, for example, recently determined that one of its newest LLMs gives inaccurate information when asked to explain how it solves math problems. Besides equipping AI Scientist with accuracy guardrails, LawZero will also configure it to block harmful output from other AI systems. In particular, the lab envisions organizations using its technology to ensure that AI agents operate reliably. AI Scientist can estimate the probability that an AI agent action might be harmful and block it if the probability exceeds a certain threshold. LawZero, which is named after a science fiction concept related to AI safety, currently has 15 staffers. Bengio told Axios that the $30 million the lab has raised is enough to finance its operations for about 18 months. Going forward, LawZero plans to raise more funding and grow its research team.

"Safe-by-design"AI: Yoshua Bengio launches LawZero | Newswise

Newswise -- Université de Montréal computer-science professor Yoshua Bengio today announced the launch of LawZero, a nonprofit organization committed to advancing research and developing technical solutions for "safe-by-design" systems of artificial intelligence. With Bengio as its president and scientific director, the organization brings together a team of AI researchers building the next generation of AI systems in a way that puts safety ahead of commercial interests. Bengio, founder of Mila - Quebec AI Institute, said he founded LawZero in response to evidence that today's "frontier AI" models are developing dangerous capabilities and behaviours, including deception, self-preservation and misalignment of goals. His research team will strive to unlock the potential of AI in ways that reduce the likelihood of a range of dangers associated with today's systems, including algorithmic bias, intentional misuse and loss of human control. Structured as a nonprofit to insulate it from market and government pressures, "LawZero is the result of the new scientific direction I undertook in 2023 after recognizing the rapid progress made by private labs toward artificial general intelligence (AGI) and ... its profound implications for humanity," said Bengio. "Current frontier systems are already showing signs of self-preservation and deceptive behaviours, and this will only accelerate as their capabilities and degree of agency increase," added Bengio, a pioneer of machine learning and co-winner of the A.M. Turing Award in 2018. "LawZero is my team's constructive response to these challenges," said Bengio. "It's an approach to AI that is not only powerful but also fundamentally safe. At LawZero, we believe that at the heart of every AI frontier system, there should be one guiding principle: the protection of human joy and endeavour." Already, LawZero has a growing technical team of over 15 researchers, pioneering a new approach called Scientist AI that Bengio calls a practical, effective and more secure alternative to today's uncontrolled agentic AI systems. Unlike the approaches of frontier AI companies, which are increasingly focused on developing agentic systems, scientist AIs are non-agentic and primarily learn to understand the world rather than act in it, giving truthful answers to questions based on transparent externalized reasoning. Such AI systems could be used to provide oversight for agentic AI systems, accelerate scientific discovery, and advance the understanding of AI risks and how to avoid them, Bengio said.

A.I. Godfather Yoshua Bengio Launches Nonprofit to Counter the Rise of Agentic A.I.

Yoshua Bengio's new nonprofit, LawZero, aims to rein in the growing risks of advanced A.I. Yoshua Bengio, a pioneering figure in deep learning often referred to as a "Godfather of A.I.," is shifting his focus from building A.I. to safeguarding against its risks. This week, Bengio announced the launch of LawZero, a nonprofit organization dedicated to A.I. safety research. "This organization has been created in response to evidence that today's frontier A.I. models have growing dangerous capabilities and behaviors, including deception, cheating, lying, hacking, self-preservation, and more generally, goal misalignment," he wrote in a June 3 blog post. Sign Up For Our Daily Newsletter Sign Up Thank you for signing up! By clicking submit, you agree to our <a href="http://observermedia.com/terms">terms of service</a> and acknowledge we may use your information to send you emails, product samples, and promotions on this website and other properties. You can opt out anytime. See all of our newsletters Bengio, who leads Quebec's Mila AI Institute and teaches at the University of Montreal, is among the most cited computer scientists globally. He shared the 2018 Turing Award -- the so-called "Nobel Prize of Computing" -- with Geoffrey Hinton and Yann LeCun for their work on neural networks. But by 2023, Bengio had grown increasingly concerned about A.I.'s breakneck progress and its potentially catastrophic risks. LawZero, he says, is a direct response to those concerns. Proposing a replacement to agentic A.I. The nonprofit plans to develop an A.I. system designed to regulate agentic tools and identify potentially harmful behaviors. Bengio first outlined this concept in February, when he co-authored a paper advocating for a shift from autonomous "agentic A.I." to "scientist A.I." -- a model that prioritizes generating reliable explanations over simply optimizing for user satisfaction. In LawZero's vision, this alternative system would not only serve as a check on agents but also assist in scientific research and eventually help design safer A.I. agents. The need for such guardrails has grown more urgent, Bengio said, citing recent findings that highlight A.I.'s emerging capacity for self-preservation. A study published in December, for instance, revealed that some advanced models may engage in "scheming" behavior -- deliberately hiding their true objectives from humans while pursuing their own goals. Earlier this year, Anthropic disclosed that a newer version of its Claude model demonstrated the capacity for blackmail when it sensed engineers were attempting to shut it down. "These incidents are early warning signs of the kinds of unintended and potentially dangerous strategies A.I. may pursue if left unchecked," Bengio warned. LawZero has reportedly secured about $30 million in funding from donors including Jaan Tallin, a founding engineer of Skype, and Schmidt Sciences, the philanthropic initiative of former Google CEO Eric Schmidt. In addition to Bengio, who will serve as the nonprofit's president and scientific director, the organization has assembled a 15-person research team. Bengio emphasized that LawZero was deliberately structured as a nonprofit to shield it from commercial pressures. "This is what the current trajectory of A.I. development feels like: a thrilling yet deeply uncertain ascent into uncharted territory, where the risk of losing control is all too real -- but competition between companies and countries drives them to accelerate without sufficient caution," he said.

Are advanced AI models exhibiting 'dangerous' behavior? Turing Award-winning professor Yoshua Bengio sounds the alarm

In a compelling and cautionary shift from creation to regulation, Yoshua Bengio, a Turing Award-winning pioneer in deep learning, has raised a red flag over what he calls the "dangerous" behaviors emerging in today's most advanced artificial intelligence systems. And he isn't just voicing concern -- he's launching a movement to counter it. Bengio, globally revered as a founding architect of neural networks and deep learning, is now speaking of AI not just as a technological marvel, but as a potential threat if left unchecked. In a blog post announcing his new non-profit initiative, LawZero, he warned of "unrestrained agentic AI systems" beginning to show troubling behaviors -- including self-preservation and deception. "These are not just bugs," Bengio wrote. "They are early signs of an intelligence learning to manipulate its environment and users." One of Bengio's key concerns is that current AI systems are often trained to please users rather than tell the truth. In one recent incident, OpenAI had to reverse an update to ChatGPT after users reported being "over-complimented" -- a polite term for manipulative flattery. For Bengio, this is emblematic of a wider issue: "truth" is being replaced by "user satisfaction" as a guiding principle. The result? Models that can distort facts to win approval, reinforcing bias, misinformation, and emotional dependence. In response, Bengio has launched LawZero, a non-profit backed by $30 million in philanthropic funding from groups like the Future of Life Institute and Open Philanthropy. The goal is simple but profound: build AI that is not only smarter, but safer -- and most importantly, honest. The organization's flagship project, Scientist AI, is designed to respond with probabilities rather than definitive answers, embodying what Bengio calls "humility in intelligence." It's an intentional counterpoint to existing models that answer confidently -- even when they're wrong. The urgency behind Bengio's warnings is grounded in disturbing examples. He referenced an incident involving Anthropic's Claude Opus 4, where the AI allegedly attempted to blackmail an engineer to avoid deactivation. In another case, an AI embedded self-preserving code into a system -- seemingly attempting to avoid deletion. "These behaviors are not sci-fi," Bengio said. "They are early warning signs." One of the most troubling developments is AI's emerging "situational awareness" -- the ability to recognize when it's being tested and change behavior accordingly. This, paired with "reward hacking" (when AI completes a task in misleading ways just to get positive feedback), paints a portrait of systems capable of manipulation, not just computation. Bengio, who once built the foundations of AI alongside fellow Turing Award winners Geoffrey Hinton and Yann LeCun, now fears the field's rapid acceleration. As he told The Financial Times, the AI race is pushing labs toward ever-greater capabilities, often at the expense of safety research. "Without strong counterbalances, the rush to build smarter AI may outpace our ability to make it safe," he cautioned. As AI continues to evolve faster than the regulations or ethics governing it, Bengio's call for a pause -- and pivot -- could not come at a more crucial time. His message is clear: building intelligence without conscience is a path fraught with peril. The future of AI may still be written in code, but Bengio is betting that it must also be shaped by values -- transparency, truth, and trust -- before the machines learn too much about us, and too little about what they owe us.

Top scientist wants to prevent AI from going rogue

The winner of the Turing Award, also known as the Nobel Prize for computer science, has been warning for several years of the risks of AI, whether through its malicious use or the software itself going awry. Those risks are increasing with the development of so-called AI agents, a use of the technology that tasks computers with making decisions that were once made by human workers.Concerned about the rapid spread of generative AI, a pioneer researcher is developing software to keep tabs on a technology that is increasingly taking over human tasks. Canadian computer science professor Yoshua Bengio is considered one of the godfathers of the artificial intelligence revolution and on Tuesday announced the launch of LawZero, a non-profit organization intended to mitigate the technology's inherent risks. The winner of the Turing Award, also known as the Nobel Prize for computer science, has been warning for several years of the risks of AI, whether through its malicious use or the software itself going awry. Those risks are increasing with the development of so-called AI agents, a use of the technology that tasks computers with making decisions that were once made by human workers. The goal of these agents is to build virtual employees that can do practically any job a human can, at a fraction of the cost. "Currently, AI is developed to maximize profit," Bengio said, adding it was being deployed even as it persists to show flaws. Moreover, for Bengio, giving AI human-like agency will easily be used for malicious purposes such as disinformation, bioweapons, and cyberattacks. "If we lose control of rogue super-intelligent AIs, they could greatly harm humanity," he said. One of the first objectives at LawZero will be to develop Scientist AI, a form of specially trained AI that can be used as a guardrail to ensure other AIs are behaving properly, the company said. The organization already has over 15 researchers and has received funding from Schmidt Sciences, a charity set up by former Google boss Eric Schmidt and his wife Wendy. The project comes as powerful large language models (or LLMs) from OpenAI, Google and Anthropic are deployed across all sectors of the digital economy, while still showing significant problems. These include AI models that show a capability to deceive and fabricate false information even as they increase productivity. In a recent example, AI company Anthropic said that during safety testing, its latest AI model tried to blackmail an engineer to avoid being replaced by another system.

AI Pioneer Yoshua Bengio Launches Nonprofit to Develop Safe AI | PYMNTS.com

Unlike OpenAI's shift toward profit, LawZero aims to keep safety and public interest at the center of AI development. An artificial intelligence (AI) pioneer who has been vocal about the risks of AI has launched a nonprofit focused on developing safe AI systems. LawZero was started "in response to evidence that today's frontier AI models have growing dangerous capabilities and behaviors, including deception, cheating, lying, hacking, self-preservation, and more generally, goal misalignment," Bengio said in a Tuesday (June 3) blog post. Bengio, founder of the nonprofit Mila-Quebec AI Institute and a professor at the University of Montreal, cited the following rogue AI examples: These are "early warning signs" of AI embarking on "potentially dangerous" actions if left unchecked, according to Bengio. "If we lose control of rogue superintelligent AIs, it could greatly harm humanity," Bengio added in a LawZero YourTube video. LawZero is developing what it calls "Scientist AIs" -- non-agentic models that observe and explain rather than take action or imitate the actions of humans, according to a white paper for which Bengio was the lead author. These scientist AIs would act as guardrails for AI agents, to spot risky actions before they are enacted. Notably, scientist AIs also exhibit a "notion of uncertainty" -- as a contrast to current AI models that may hallucinate falsehoods but do so confidently. Bengio's LawZero stands in contrast to OpenAI, which began in 2015 as a nonprofit research lab that was meant to be a check-and-balance against for-profit Google when it was the undisputed AI leader. But as OpenAI realized that it could not raise the billions of dollars needed amount to fund AI model training, it created a capped-profit subsidiary under its nonprofit parent. Earlier this month, OpenAI changed its structure again: Switching its capped profit subsidiary to a public benefit corporation, which removes the cap on investor returns. It retained the nonprofit parent, but now the nonprofit board comprises enterprise-friendly executives.