Global Consensus Emerges on AI Safety Research Priorities

Singapore's Vision for AI Safety Bridges the US-China Divide

In a rare moment of global consensus, AI researchers from the US, Europe, and Asia came together in Singapore to form a plan for researching AI risks. The government of Singapore released a blueprint today for global collaboration on artificial intelligence safety following a meeting of AI researchers from the US, China, and Europe. The document lays out a shared vision for working on AI safety through international cooperation rather than competition. "Singapore is one of the few countries on the planet that gets along well with both East and West," says Max Tegmark, a scientist at MIT who helped convene the meeting of AI luminaries last month. "They know that they're not going to build [artificial general intelligence] themselves -- they will have it done to them -- so it is very much in their interests to have the countries that are going to build it talk to each other." The countries thought most likely to build AGI are, of course, the US and China -- and yet those nations seem more intent on outmaneuvering each other than working together. In January, after Chinese startup DeepSeek released a cutting-edge model, President Trump called it "a wakeup call for our industries" and said the US needed to be "laser-focused on competing to win." The Singapore Consensus on Global AI Safety Research Priorities calls for researchers to collaborate in three key areas: studying the risks posed by frontier AI models, exploring safer ways to build those models, and developing methods for controlling the behavior of the most advanced AI systems. The consensus was developed at a meeting held on April 26 alongside the International Conference on Learning Representations (ICLR), a premier AI event held in Singapore this year. Researchers from OpenAI, Anthropic, Google DeepMind, xAI, and Meta all attended the AI safety event, as did academics from institutions including MIT, Stanford, Tsinghua, and the Chinese Academy of Sciences. Experts from AI safety institutes in the US, UK, France, Canada, China, Japan and Korea also participated. "In an era of geopolitical fragmentation, this comprehensive synthesis of cutting-edge research on AI safety is a promising sign that the global community is coming together with a shared commitment to shaping a safer AI future," Xue Lan, dean of Tsinghua University, said in a statement. The development of increasingly capable AI models, some of which have surprising abilities, has caused researchers to worry about a range of risks. While some focus on near-term harms including problems caused by biased AI systems or the potential for criminals to harness the technology, a significant number believe that AI may pose an existential threat to humanity as it begins to outsmart humans across more domains. These researchers, sometimes referred to as "AI doomers," worry that models may deceive and manipulate humans in order to pursue their own goals. The potential of AI has also stoked talk of an arms race between the US, China, and other powerful nations. The technology is viewed in policy circles as critical to economic prosperity and military dominance, and many governments have sought to stake out their own visions and regulations governing how it should be developed.

100 leading AI scientists map route to more 'trustworthy, reliable, secure' AI

The debate over the risks and harms of artificial intelligence often focuses on what governments can or should do. However, just as important are the choices that AI researchers themselves make. This week, in Singapore, more than 100 scientists from around the world proposed guidelines for how researchers should approach making AI more "trustworthy, reliable, and secure." Also: A few secretive AI companies could crush free society, researchers warn The recommendations come at a time when the giants of generative AI, such as OpenAI and Google, have increasingly reduced disclosures about their AI models, so the public knows less and less about how the work is conducted. The guidelines grew out of an exchange among the scholars last month in Singapore, in conjunction with one of the most prestigious conferences on AI, the International Conference on Learning Representations -- the first time a major AI conference has taken place in Asia. The document, "The Singapore Consensus on Global AI Safety Research Priorities," is posted at the website of the Singapore Conference on AI, a second AI conference taking place this week in Singapore. Among the luminaries who helped to draft the Singapore Consensus are Yoshua Bengio, founder of Canada's AI institute, MILA; Stuart Russell, U.C. Berkeley distinguished professor of computer science, and an expert on "human-centered AI"; Max Tegmark, head of the UK-based think tank The Future of Life Institute; and representatives from the Massachusetts Institute of Technology, Google's DeepMind unit, Microsoft, the National University of Singapore, and China's Tsinghua University and National Academy of Sciences, among others. To make the case that research must have guidelines, Singapore's Minister for Digital Development and Information, Josephine Teo, in presenting the work, noted that people can't vote for what kind of AI they want. "In democracies, general elections are a way for citizens to choose the party that forms the government and to make decisions on their behalf," said Teo. "But in AI development, citizens do not get to make a similar choice. However democratising we say the technology is, citizens will be at the receiving end of AI's opportunities and challenges, without much say over who shapes its trajectory." Also: Google's Gemini continues the dangerous obfuscation of AI technology The paper lays out three categories researchers should consider: How to identify risks, how to build AI systems in such a way as to avoid risks, and how to maintain control over AI systems, meaning, ways to monitor and intervene in the case of concerns about those AI systems. "Our goal is to enable more impactful R&D efforts to rapidly develop safety and evaluation mechanisms and foster a trusted ecosystem where AI is harnessed for the public good," the authors write in the preface to the report. "The motivation is clear: no organisation or country benefits when AI incidents occur or malicious actors are enabled, as the resulting harm would damage everyone collectively." On the first score, assessing potential risks, the scholars advised the development of "metrology," the measurement of potential harm. They write that there is a need for "quantitative risk assessment tailored to AI systems to reduce uncertainty and the need for large safety margins." There's a need to allow outside parties to monitor AI research and development for risk, the scholars note, with a balance on protecting corporate IP. That includes developing "secure infrastructure that enables thorough evaluation while protecting intellectual property, including preventing model theft." Also: Stuart Russell: Will we choose the right objective for AI before it destroys us all? The development section concerns how to make AI trustworthy, reliable, and secure "by design." To do so, there's a need to develop "technical methods" that can specify what's intended from an AI program and also outline what should not happen -- the "undesired side effects" -- the scholars write. The actual training of neural nets then needs to be advanced in such a way that the resulting AI programs are "guaranteed to meet their specifications," they write. That includes parts of training that focus on, for example, "reducing confabulation" (often known as hallucinations) and "increasing robustness against tampering," such as cracking an LLM with malicious prompts. Last, the control section of the paper covers both how to extend current computer security measures and how to develop new techniques to avoid runaway AI. For example, conventional computer control, such as off-switches and override protocols, needs to be extended to handle AI programs. Scientists also need to design "new techniques for controlling very powerful AI systems that may actively undermine attempts to control them." The paper is ambitious, which is appropriate given rising concern about the risk from AI as it connects to more and more computer systems, such as agentic AI. Also: Multimodal AI poses new safety risks, creates CSEM and weapons info As the scientists acknowledge in the introduction, research on safety won't be able to keep up with the rapid pace of AI unless more investment is made. "Given that the state of science today for building trustworthy AI does not fully cover all risks, accelerated investment in research is required to keep pace with commercially driven growth in system capabilities," write the authors. Writing in Time magazine, Bengio echoes the concerns about runaway AI systems. "Recent scientific evidence also demonstrates that, as highly capable systems become increasingly autonomous AI agents, they tend to display goals that were not programmed explicitly and are not necessarily aligned with human interests," writes Bengio. "I'm genuinely unsettled by the behavior unrestrained AI is already demonstrating, in particular self-preservation and deception."

'Rival governments don't want nuclear AI blowups,' says AI expert

A new paper released on Thursday, presenting the findings of a global conference held in Singapore in April, has identified how AI can be made safer. The last global gathering on artificial intelligence (AI) at the Paris AI Action Summit in February saw countries divided, notably after the US and UK refused to sign a joint declaration for AI that is "open, inclusive, transparent, ethical, safe, secure, and trustworthy". AI experts at the time criticised the declaration for not going far enough and being "devoid of any meaning," the reason countries cited for not signing the pact, as opposed to their being against AI safety. The next global AI summit will be held in India next year, but rather than wait until then, Singapore's government held a conference called the International Scientific Exchange on AI Safety on April 26. "Paris [AI Summit] left a misguided impression that people don't agree about AI safety," said Max Tegmark, MIT professor and contributor to the Singapore report. "The Singapore government was clever to say yes, there is an agreement," he told Euronews Next. Representatives from leading AI companies, such as OpenAI, Meta, Google DeepMind, and Anthropic, as well as leaders from 11 countries, including the US, China, and the EU, attended. The result of the conference was published in a paper released on Thursday called 'The Singapore Consensus on Global AI Safety Research Priorities'. The document lists research proposals to ensure that AI does not become dangerous to humanity. It identifies three aspects to promote a safe AI: assessing, developing trustworthiness, and controlling AI systems, which include large language models (LLMs), multimodal models that can work with multiple types of data, often including text, images, video, and lastly, AI agents. Assessing AI The main research that the document argues should be assessed is the development of risk thresholds to determine when intervention is needed, techniques for studying current impacts and forecasting future implications, and methods for rigorous testing and evaluation of AI systems. Some of the key areas of research listed include improving the validity and precision of AI model assessments and finding methods for testing dangerous behaviours, which include scenarios where AI operates outside human control. Developing trustworthy, secure, and reliable AI The paper calls for a definition of boundaries between acceptable and unacceptable behaviours. It also says that when building AI systems, they should be developed with truthful and honest systems and datasets. And once built, these AI systems should be checked to ensure they meet agreed safety standards, such as tests against jailbreaking. Control The final area the paper advocates for is the control and societal resilience of AI systems. This includes monitoring, kill switches, and non-agentic AI serving as guardrails for agentic systems. It also calls for human-centric oversight frameworks. As for societal resilience, the paper said that infrastructure against AI-enabled disruptions should be strengthened, and it argued that coordination mechanisms for incident responses should be developed. 'Not in their interest' The release of the report comes as the geopolitical race for AI intensifies and AI companies thrash out their latest models to beat their competition. However, Xue Lan, Dean of Tsinghua University, who attended the conference, said: "In an era of geopolitical fragmentation, this comprehensive synthesis of cutting-edge research on AI safety is a promising sign that the global community is coming together with a shared commitment to shaping a safer AI future". Tegmark added that there is a consensus for AI safety between governments and tech firms, as it is in everyone's interest. "OpenAI, Antropic, and all these companies sent people to the Singapore conference; they want to share their safety concerns, and they don't have to share their secret sauce," he said. "Rival governments also don't want nuclear blow-ups in opposing countries, it's not in their interest," he added. Tegmark hopes that before the next AI summit in India, governments will treat AI like any other powerful tech industry, such as biotech, whereby there are safety standards in each country and new drugs are required to pass certain trials. "I'm feeling much more optimistic about the next summit now than after Paris," Tegmark said.

Researchers reboot push for AI safety after Paris summit bust

Experts researching threats stemming from artificial intelligence agreed on key work areas needed to contain dangers like loss of human control or easily accessible bioweapons in a report published Thursday. In "AI 2027", a widely-read scenario recently published online by a small group of researchers, competition between the United States and China drives Washington to cede control over its economy and military to a rogue AI, ultimately resulting in human extinction.Experts researching threats stemming from artificial intelligence agreed on key work areas needed to contain dangers like loss of human control or easily accessible bioweapons in a report published Thursday. Many safety-focused scientists were disappointed by February's Paris AI summit, where the French hosts largely left aside threats to home in on hoped-for economic boons. But "the mood was exactly the opposite of Paris" at a gathering of experts in Singapore in late April, said MIT researcher and conference organiser Max Tegmark, president of the Future of Life Institute that charts existential risks. "A lot of people came up to me and said that they had gotten their mojo back now... there's hope again," he told AFP. In a report put together at the conference, the experts name three overlapping work areas to focus on faced with ever-more-capable AIs: assessing risk from AI and its applications; developing AI that is safe and trustworthy by design; and monitoring deployed AI -- ready to intervene if alert signals flash. There is "global convergence around the technical challenges in AI safety", said leading researcher Yoshua Bengio, who helped compile the "Singapore Consensus on Global AI Safety Research Priorities" report. "We have work to do that everybody agrees should be done. The Americans and the Chinese agree," Tegmark added. The AI safety community can be a gloomy place, with dire predictions of AI escaping human control altogether or proferring step-by-step instructions to build biological weapons -- even as tech giants plough hundreds of billions into building more powerful intelligences. In "AI 2027", a widely-read scenario recently published online by a small group of researchers, competition between the United States and China drives Washington to cede control over its economy and military to a rogue AI, ultimately resulting in human extinction. Online discussions pore over almost weekly hints that the latest AI models from major companies such as OpenAI or Anthropic could be trying to outwit researchers probing their capabilities and inner workings, which remain largely impenetrable even to their creators. Next year's governmental AI summit in India is widely expected to echo the optimistic tone of Paris. But Tegmark said that even running in parallel to politicians' quest for economic payoffs, experts' research can influence policy towards enforcing safety on those building and deploying AI. "The easiest way to get the political will is to do the nerd research. We've never had a nuclear winter. We didn't need to have one in order for (Soviet leader Mikhail) Gorbachev and (US President Ronald) Reagan to take it seriously" -- and agree on nuclear arms restraint, he said. Researchers' conversations in Singapore were just as impactful as the Paris summit was, "but with the impact going in a very, very different direction," Tegmark said.