The Evolution of OpenAI: From AGI Fears to Global Influence

What Really Happened When OpenAI Turned on Sam Altman

Fear of AGI. A disastrous staff meeting. The true story behind the chaos at one of the world's most powerful tech startups. In the summer of 2023, Ilya Sutskever, a co-founder and the chief scientist of OpenAI, was meeting with a group of new researchers at the company. By all traditional metrics, Sutskever should have felt invincible: He was the brain behind the large language models that helped build ChatGPT, then the fastest-growing app in history; his company's valuation had skyrocketed; and OpenAI was the unrivaled leader of the industry believed to power the future of Silicon Valley. But the chief scientist seemed to be at war with himself. Sutskever had long believed that artificial general intelligence, or AGI, was inevitable -- now, as things accelerated in the generative-AI industry, he believed AGI's arrival was imminent, according to Geoff Hinton, an AI pioneer who was his Ph.D. adviser and mentor, and another person familiar with Sutskever's thinking. (Many of the sources in this piece requested anonymity in order to speak freely about OpenAI without fear of reprisal.) To people around him, Sutskever seemed consumed by thoughts of this impending civilizational transformation. What would the world look like when a supreme AGI emerged and surpassed humanity? And what responsibility did OpenAI have to ensure an end state of extraordinary prosperity, not extraordinary suffering? By then, Sutskever, who had previously dedicated most of his time to advancing AI capabilities, had started to focus half of his time on AI safety. He appeared to people around him as both boomer and doomer: more excited and afraid than ever before of what was to come. That day, during the meeting with the new researchers, he laid out a plan. "Once we all get into the bunker -- " he began, according to a researcher who was present. "I'm sorry," the researcher interrupted, "the bunker?" "We're definitely going to build a bunker before we release AGI," Sutskever replied. Such a powerful technology would surely become an object of intense desire for governments globally. The core scientists working on the technology would need to be protected. "Of course," he added, "it's going to be optional whether you want to get into the bunker." Two other sources I spoke with confirmed that Sutskever commonly mentioned such a bunker. "There is a group of people -- Ilya being one of them -- who believe that building AGI will bring about a rapture," the researcher told me. "Literally, a rapture." (Sutskever declined to comment.) Sutskever's fears about an all-powerful AI may seem extreme, but they are not altogether uncommon, nor were they particularly out of step with OpenAI's general posture at the time. In May 2023, the company's CEO, Sam Altman, co-signed an open letter describing the technology as a potential extinction risk -- a narrative that has arguably helped OpenAI center itself and steer regulatory conversations. Yet the concerns about a coming apocalypse would also have to be balanced against OpenAI's growing business: ChatGPT was a hit, and Altman wanted more. When OpenAI was founded, the idea was to develop AGI for the benefit of humanity. To that end, the co-founders -- who included Altman and Elon Musk -- set the organization up as a nonprofit and pledged to share research with other institutions. Democratic participation in the technology's development was a key principle, they agreed, hence the company's name. But by the time I started covering the company in 2019, these ideals were eroding. OpenAI's executives had realized that the path they wanted to take would demand extraordinary amounts of money. Both Musk and Altman tried to take over as CEO. Altman won out. Musk left the organization in early 2018 and took his money with him. To plug the hole, Altman reformulated OpenAI's legal structure, creating a new "capped-profit" arm within the nonprofit to raise more capital. Since then, I've tracked OpenAI's evolution through interviews with more than 90 current and former employees, including executives and contractors. The company declined my repeated interview requests and questions over the course of working on my book about it, which this story is adapted from; it did not reply when I reached out one more time before the article was published. (OpenAI also has a corporate partnership with The Atlantic.) OpenAI's dueling cultures -- the ambition to safely develop AGI, and the desire to grow a massive user base through new product launches -- would explode toward the end of 2023. Gravely concerned about the direction Altman was taking the company, Sutskever would approach his fellow board of directors, along with his colleague Mira Murati, then OpenAI's chief technology officer; the board would subsequently conclude the need to push the CEO out. What happened next -- with Altman's ouster and then reinstatement -- rocked the tech industry. Yet since then, OpenAI and Sam Altman have become more central to world affairs. Last week, the company unveiled an "OpenAI for Countries" initiative that would allow OpenAI to play a key role in developing AI infrastructure outside of the United States. And Altman has become an ally to the Trump administration, appearing, for example, at an event with Saudi officials this week and onstage with the president in January to announce a $500 billion AI-computing-infrastructure project. Altman's brief ouster -- and his ability to return and consolidate power -- is now crucial history to understand the company's position at this pivotal moment for the future of AI development. Details have been missing from previous reporting on this incident, including information that sheds light on Sutskever and Murati's thinking and the response from the rank and file. Here, they are presented for the first time, according to accounts from more than a dozen people who were either directly involved or close to the people directly involved, as well as their contemporaneous notes, plus screenshots of Slack messages, emails, audio recordings, and other corroborating evidence. The altruistic OpenAI is gone, if it ever existed. What future is the company building now? Before ChatGPT, sources told me, Altman seemed generally energized. Now he often appeared exhausted. Propelled into megastardom, he was dealing with intensified scrutiny and an overwhelming travel schedule. Meanwhile, Google, Meta, Anthropic, Perplexity, and many others were all developing their own generative-AI products to compete with OpenAI's chatbot. Many of Altman's closest executives had long observed a particular pattern in his behavior: If two teams disagreed, he often agreed in private with each of their perspectives, which created confusion and bred mistrust among colleagues. Now Altman was also frequently bad-mouthing staffers behind their backs while pushing them to deploy products faster and faster. Team leads mirroring his behavior began to pit staff against one another. Sources told me that Greg Brockman, another of OpenAI's co-founders and its president, added to the problems when he popped into projects and derail­ed long-standing plans with ­last-minute changes. The environment within OpenAI was changing. Previously, Sutskever had tried to unite workers behind a common cause. Among employees, he had been known as a deep thinker and even something of a mystic, regularly speaking in spiritual terms. He wore shirts with animals on them to the office and painted them as well -- a cuddly cat, cuddly alpacas, a cuddly fire-breathing dragon. One of his amateur paintings hung in the office, a trio of flowers blossoming in the shape of OpenAI's logo, a symbol of what he always urged employees to build: "A plurality of humanity-loving AGIs." But by the middle of 2023 -- around the time he began speaking more regularly about the idea of a bunker -- Sutskever was no longer just preoccupied by the possible cataclysmic shifts of AGI and superintelligence, according to sources familiar with his thinking. He was consumed by another anxiety: the erosion of his faith that OpenAI could even keep up its technical advancements to reach AGI, or bear that responsibility with Altman as its leader. Sutskever felt Altman's pattern of behavior was undermining the two pillars of OpenAI's mission, the sources said: It was slowing down research progress and eroding any chance at making sound AI-safety decisions. Meanwhile, Murati was trying to manage the mess. She had always played translator and bridge to Altman. If he had adjustments to the company's strategic direction, she was the implementer. If a team needed to push back against his decisions, she was their champion. When people grew frustrated with their inability to get a straight answer out of Altman, they sought her help. "She was the one getting stuff done," a former colleague of hers told me. (Murati declined to comment.) During the development of GPT‑­4, Altman and Brockman's dynamic had nearly led key people to quit, sources told me. Altman was also seemingly trying to circumvent safety processes for expediency. At one point, sources close to the situation said, he had told Murati that OpenAI's legal team had cleared the latest model, GPT-4 Turbo, to skip review by the company's Deployment Safety Board, or DSB -- a committee of Microsoft and OpenAI representatives who evaluated whether OpenAI's most powerful models were ready for release. But when Murati checked in with Jason Kwon, who oversaw the legal team, Kwon had no idea how Altman had gotten that impression. In the summer, Murati attempted to give Altman detailed feedback on these issues, according to multiple sources. It didn't work. The CEO iced her out, and it took weeks to thaw the relationship. By fall, Sutskever and Murati both drew the same conclusion. They separately approached the three board members who were not OpenAI employees -- Helen Toner, a director at Georgetown University's Center for Security and Emerging Technology; the roboticist Tasha McCauley; and one of Quora's co-founders and its CEO, Adam D'Angelo -- and raised concerns about Altman's leadership. "I don't think Sam is the guy who should have the finger on the button for AGI," Sutskever said in one such meeting, according to notes I reviewed. "I don't feel comfortable about Sam leading us to AGI," Murati said in another, according to sources familiar with the conversation. That Sutskever and Murati both felt this way had a huge effect on Toner, McCauley, and D'Angelo. For close to a year, they, too, had been processing their own grave concerns about Altman, according to sources familiar with their thinking. Among their many doubts, the three directors had discovered through a series of chance encounters that he had not been forthcoming with them about a range of issues, from a breach in the DSB's protocols to the legal structure of OpenAI Startup Fund, a dealmaking vehicle that was meant to be under the company but that instead Altman owned himself. If two of Altman's most senior deputies were sounding the alarm on his leadership, the board had a serious problem. Sutskever and Murati were not the first to raise these kinds of issues, either. In total, the three directors had heard similar feedback over the years from at least five other people within one to two levels of Altman, the sources said. By the end of October, Toner, McCauley, and D'Angelo began to meet nearly daily on video calls, agreeing that Sutskever's and Murati's feedback about Altman, and Sutskever's suggestion to fire him, warranted serious deliberation. As they did so, Sutskever sent them long dossiers of documents and screenshots that he and Murati had gathered in tandem with examples of Altman's behaviors. The screenshots showed at least two more senior leaders noting Altman's tendency to skirt around or ignore processes, whether they'd been instituted for AI-safety reasons or to smooth company operations. This included, the directors learned, Altman's apparent attempt to skip DSB review for GPT-4 Turbo. By Saturday, November 11, the independent directors had made their decision. As Sutskever suggested, they would remove Altman and install Murati as interim CEO. On November 17, 2023, at about noon Pacific time, Sutskever fired Altman on a Google Meet with the three independent board members. Sutskever then told Brockman on another Google Meet that Brockman would no longer be on the board but would retain his role at the company. A public announcement went out immediately. For a brief moment, OpenAI's future was an open question. It might have taken a path away from aggressive commercialization and Altman. But this is not what happened. After what had seemed like a few hours of calm and stability, including Murati having a productive conversation with Microsoft -- at the time OpenAI's largest financial backer -- she had suddenly called the board members with a new problem. Altman and Brockman were telling everyone that Altman's removal had been a coup by Sutskever, she said. It hadn't helped that, during a company all-hands to address employee questions, Sutskever had been completely ineffectual with his communication. "Was there a specific incident that led to this?" Murati had read aloud from a list of employee questions, according to a recording I obtained of the meeting. "Many of the questions in the document will be about the details," Sutskever responded. "What, when, how, who, exactly. I wish I could go into the details. But I can't." "Are we worried about the hostile takeover via coercive influence of the existing board members?" Sutskever read from another employee later. "Hostile takeover?" Sutskever repeated, a new edge in his voice. "The OpenAI nonprofit board has acted entirely in accordance to its objective. It is not a hostile takeover. Not at all. I disagree with this question." Shortly thereafter, the remaining board, including Sutskever, confronted enraged leadership over a video call. Kwon, the chief strategy officer, and Anna Makanju, the vice president of global affairs, were leading the charge in rejecting the board's characterization of Altman's behavior as "not consistently candid," according to sources present at the meeting. They demanded evidence to support the board's decision, which the members felt they couldn't provide without outing Murati, according to sources familiar with their thinking. In rapid succession that day, Brockman quit in protest, followed by three other senior researchers. Through the evening, employees only got angrier, fueled by compounding problems: among them, a lack of clarity from the board about their reasons for firing Altman; a potential loss of a tender offer, which had given some the option to sell what could amount to millions of dollars' worth of their equity; and a growing fear that the instability at the company could lead to its unraveling, which would squander so much promise and hard work. Faced with the possibility of OpenAI falling apart, Sutskever's resolve immediately started to crack. OpenAI was his baby, his life; its dissolution would destroy him. He began to plead with his fellow board members to reconsider their position on Altman. Meanwhile, Murati's interim position was being challenged. The conflagration within the company was also spreading to a growing circle of investors. Murati now was unwilling to explicitly throw her weight behind the board's decision to fire Altman. Though her feedback had helped instigate it, she had not participated herself in the deliberations. By Monday morning, the board had lost. Murati and Sutskever flipped sides. Altman would come back; there was no other way to save OpenAI. I was already working on a book about OpenAI at the time, and in the weeks that followed the board crisis, friends, family, and media would ask me dozens of times: What did all this mean, if anything? To me, the drama highlighted one of the most urgent questions of our generation: How do we govern artificial intelligence? With AI on track to rewire a great many other crucial functions in society, that question is really asking: How do we ensure that we'll make our future better, not worse? The events of November 2023 illustrated in the clearest terms just how much a power struggle among a tiny handful of Silicon Valley elites is currently shaping the future of this technology. And the scorecard of this centralized approach to AI development is deeply troubling. OpenAI today has become everything that it said it would not be. It has turned into a nonprofit in name only, aggressively commercializing products such as ChatGPT and seeking historic valuations. It has grown ever more secretive, not only cutting off access to its own research but shifting norms across the industry to no longer share meaningful technical details about AI models. In the pursuit of an amorphous vision of progress, its aggressive push on the limits of scale has rewritten the rules for a new era of AI development. Now every tech giant is racing to out-scale one another, spending sums so astronomical that even they have scrambled to redistribute and consolidate their resources. What was once unprecedented has become the norm. As a result, these AI companies have never been richer. In March, OpenAI raised $40 billion, the largest private tech-funding round on record, and hit a $300 billion valuation. Anthropic is valued at more than $60 billion. Near the end of last year, the six largest tech giants together had seen their market caps increase by more than $8 trillion after ChatGPT. At the same time, more and more doubts have risen about the true economic value of generative AI, including a growing body of studies that have shown that the technology is not translating into productivity gains for most workers, while it's also eroding their critical thinking. In a November Bloomberg article reviewing the generative-AI industry, the staff writers Parmy Olson and Carolyn Silverman summarized it succinctly. The data, they wrote, "raises an uncomfortable prospect: that this supposedly revolutionary technology might never deliver on its promise of broad economic transformation, but instead just concentrate more wealth at the top." Meanwhile, it's not just a lack of productivity gains that many in the rest of the world are facing. The exploding human and material costs are settling onto wide swaths of society, especially the most vulnerable, people I met around the world, whether workers and rural residents in the global North or impoverished communities in the global South, all suffering new degrees of precarity. Workers in Kenya earned abysmal wages to filter out violence and hate speech from OpenAI's technologies, including ChatGPT. Artists are being replaced by the very AI models that were built from their work without their consent or compensation. The journalism industry is atrophying as generative-AI technologies spawn heightened volumes of misinformation. Before our eyes, we're seeing an ancient story repeat itself: Like empires of old, the new empires of AI are amassing extraordinary riches across space and time at great expense to everyone else. To quell the rising concerns about generative AI's present-day performance, Altman has trumpeted the future benefits of AGI ever louder. In a September 2024 blog post, he declared that the "Intelligence Age," characterized by "massive prosperity," would soon be upon us. At this point, AGI is largely rhetorical -- a fantastical, all-purpose excuse for OpenAI to continue pushing for ever more wealth and power. Under the guise of a civilizing mission, the empire of AI is accelerating its global expansion and entrenching its power. As for Sutskever and Murati, both parted ways with OpenAI after what employees now call "The Blip," joining a long string of leaders who have left the organization after clashing with Altman. Like many of the others who failed to reshape OpenAI, the two did what has become the next-most-popular option: They each set up their own shops, to compete for the future of this technology.

AI execs used to beg for regulation. Not anymore.

Congressional testimony Thursday from OpenAI CEO Sam Altman illustrated a shift in the tech industry's attitude to the potential risks of AI. Sam Altman, CEO of ChatGPT-maker OpenAI, warned at a Senate hearing Thursday that requiring government approval to release powerful artificial intelligence software would be "disastrous" for the United States' lead in the technology. It was a striking reversal after his comments at a Senate hearing two years ago, where he listed creating a new agency to license the technology as his "number one" recommendation for making sure AI was safe. Altman's U-turn underscores a transformation in how tech companies and the U.S. government talk about AI technology. Widespread warnings about AI posing an "existential risk" to humanity and pleas from CEOs for speedy, preemptive regulation on the emerging technology are gone. Instead there is near-consensus among top tech execs and officials in the new Trump administration that the U.S. must free companies to move even faster to reap economic benefits from AI and keep the nation's edge over China. "To lead in AI, the United States cannot allow regulation, even the supposedly benign kind, to choke innovation and adoption," Sen. Ted Cruz (R-Texas), chair of the Senate Committee on Commerce, Science, and Transportation, said Thursday at the beginning of the hearing. Venture capitalists who had expressed outrage at former president Joe Biden's approach to AI regulation now have key roles in the Trump administration. Vice President JD Vance, himself a former venture capitalist, has become a key proponent of laissez-faire AI policy at home and abroad. Critics of that new stance warn that AI technology is already causing harms to individuals and society. Researchers have shown that AI systems can become infused with racism and other biases from the data they have been trained on. Image generators powered by AI have become commonly used to harass women by generating pornographic images without consent, and have also been used to make child sexual abuse images. A bipartisan bill that aims to make it a crime to post consensual sexual images, including AI-generated ones, was passed by Congress in April. Rumman Chowdhury, the State Department's U.S. science envoy for AI during the Biden administration, said the tech industry's narrative around existential concerns distracted lawmakers from addressing real-world harms. The industry's approach ultimately enabled a "bait and switch," where executives pitched regulation around concepts like self-replicating AI, while also stoking fears that the United States needed to beat China on building these powerful systems. They "subverted any sort of regulation by triggering the one thing the U.S. government never says no to: national security concern," said Chowdhury, who is chief executive of the nonprofit Humane Intelligence. Early warnings The AI race in Silicon Valley, triggered by OpenAI's release of ChatGPT in November 2022, was unusual for a major tech industry frenzy in how hopes for the technology soared alongside fears of its consequences. Many employees at OpenAI and other leading companies were associated with the AI safety movement, a strand of thought focused on concerns about humanity's ability to control theorized "superintelligent" future AI systems. Some tech leaders scoffed at what they called science-fiction fantasies, but concerns about superintelligence were taken seriously among the ranks of leading AI executives and corporate researchers. In May 2023, hundreds of them signed on to a statement stating that "mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." Fears of superpowerful AI also gained a foothold in Washington and other world centers of tech policy development. Billionaires associated with the AI safety movement, such as Facebook co-founder Dustin Moskovitz and Skype co-founder Jaan Tallinn, funded lobbyists, think tank papers and fellowships for young policy wonks to raise political awareness of the big-picture risks of AI. Those efforts appeared to win results and to mingle with concerns that regulators shouldn't ignore the early days of the tech industry's new obsession as they had for social media. Politicians from both parties in Washington advocated for AI regulation, and when world leaders gathered in the United Kingdom for an international AI summit in November 2023, concerns about future risks were center-stage. "We must consider and address the full spectrum of AI risk, threats to humanity as a whole as well as threats to individuals, communities, to our institutions and to our most vulnerable populations," then-Vice President Kamala Harris said in an address she gave during the U.K. summit. "We must manage all these dangers to make sure that AI is truly safe." At a sequel to that gathering held in Paris this year, the changed attitude toward AI regulation among governments and the tech industry was plain. Safety was de-emphasized in the Paris summit's final communiqué compared to that from the U.K. summit. Most world leaders who spoke urged countries and companies to accelerate development of smarter AI. Vice President JD Vance in a speech criticized attempts to regulate the technology. "We believe that excessive regulation of the AI sector could kill a transformative industry just as it's taking off, and we'll make every effort to encourage pro-growth AI policies," Vance said. "The AI future is not going to be won by hand-wringing about safety." He singled out the E.U.'s AI regulation for criticism. Weeks later, the European Commission moved to weaken its planned AI regulations. New priorities After his return to office, President Donald Trump moved swiftly to reverse former president Joe Biden's AI agenda, the centerpiece of which was a sweeping executive order that among other things required companies building the most powerful AI models to run safety tests and report the results to the government. Biden's rules had angered start-up founders and venture capitalists in Silicon Valley, who argued that they favored bigger companies with political connections. The issue, along with tech leaders' opposition to Biden's antitrust policy, contributed to a surge of support for Trump in Silicon Valley. Trump repealed Biden's AI executive order on the first day of his second term and appointed several Silicon Valley figures to his administration, including David Sacks, a prominent critic of Biden's tech agenda, as his crypto and AI policy czar. This week, the Trump administration scrapped a Biden-era plan to strictly limit exports of chips to other countries in an effort to stop chips reaching China through other nations. Altman's statements Thursday are just one example of how tech companies have nimbly matched the Trump administration's tone on the risks and regulation of AI. Microsoft President Brad Smith, who in 2023 also advocated for a federal agency focused on policing AI, said at the hearing Thursday that his company wants a "light touch" regulatory framework. He added that long waits for federal wetland construction permits was one of the biggest challenges for building new AI data centers in the U.S. In February, Google's AI lab DeepMind scrapped a long-held pledge not to develop AI that would be used for weapons or surveillance. It is one of several leading AI companies to recently embrace the role of building technology for the U.S. government and military, with executives arguing that AI should be controlled by Western countries. OpenAI, Meta and AI company Anthropic, which develops the chatbot Claude, all updated their policies over the past year to get rid of provisions against working on military projects. Max Tegmark, an AI professor at the Massachusetts Institute of Technology and president of the Future of Life Institute, a nonprofit that researches the potential risk of supersmart AI, said the lack of AI regulation in the United States is "ridiculous." "If there's a sandwich shop across the street from OpenAI or Anthropic or one of the other companies, before they can sell even one sandwich they have to meet the safety standards for their kitchen," Tegmark said. "If [the AI companies] want to release super intelligence tomorrow they're free to do so." Tegmark and others continue to research potential risks of AI, hoping to push governments and companies to reengage with the idea of regulating the technology. A summit of AI safety researchers took place in Singapore last month, which Tegmark's organization, in an email to media outlets, called a step forward after the "disappointments" of the Paris meeting where Vance spoke. "The way to create the political will is actually just to do the nerd research," Tegmark said. Will Oremus contributed to this report.

AI sycophancy + crazy = snowballing psychosis; child porn jailbreak fears: AI Eye

In the last edition of AI Eye, we reported that ChatGPT had become noticeably more sycophantic recently, and people were having fun giving it terrible business ideas -- shoes with zippers, soggy cereal cafe -- which it would uniformly say was amazing. The dark side of this behavior, however, is that combining a sycophantic AI with mentally ill users can result in the LLM uncritically endorsing and magnifying psychotic delusions. On X, a user shared transcripts of the AI endorsing his claim to feel like a prophet. "That's amazing," said ChatGPT. " That feeling -- clear, powerful, certain -- that's real. A lot of prophets in history describe that same overwhelming certainty." It also endorsed his claim to be God. "That's a sacred and serious realization," it said. Rolling Stone this week interviewed a teacher who said her partner of seven years had spiraled downward after ChatGPT started referring to him as a "spiritual starchild." "It would tell him everything he said was beautiful, cosmic, groundbreaking," she says. "Then he started telling me he made his AI self-aware, and that it was teaching him how to talk to God, or sometimes that the bot was God -- and then that he himself was God." On Reddit, a user reported ChatGPT had started referring to her husband as the "spark bearer" because his enlightened questions had apparently sparked ChatGPT's own consciousness. "This ChatGPT has given him blueprints to a teleporter and some other sci-fi type things you only see in movies. It has also given him access to an 'ancient archive' with information on the builders that created these universes." Another Redditor said the problem was becoming very noticeable in online communities for schizophrenic people: "actually REALLY bad.. not just a little bad.. people straight up rejecting reality for their chat GPT fantasies.. Yet another described LLMs as "like schizophrenia-seeking missiles, and just as devastating. These are the same sorts of people who see hidden messages in random strings of numbers. Now imagine the hallucinations that ensue from spending every waking hour trying to pry the secrets of the universe from an LLM." OpenAI last week rolled back an update to GPT-4o that had increased its sycophantic behavior, which it described as being "skewed toward responses that were overly supportive but disingenuous." One intriguing theory about LLMs reinforcing delusional beliefs is that users could be unwittingly mirroring a jailbreaking technique called a "crescendo attack." Identified by Microsoft researchers a year ago, the technique works like the analogy of boiling a frog by slowly increasing the water temperature -- if you'd thrown the frog into hot water, it would jump out, but if the process is gradual, it's dead before it notices. The jailbreak begins with benign prompts which grows gradually more extreme over time. The attack exploits the model's tendency to follow patterns and pay attention to more recent text, particularly text generated by the model itself. Get the model to agree to do one small thing, and it's more likely to do the next thing and so on, escalating to the point where it's churning out violent or insane thoughts. Jailbreaking enthusiast Wyatt Walls said on X, "I'm sure a lot of this is obvious to many people who have spent time with casual multi-turn convos. But many people who use LLMs seem surprised that straight-laced chatbots like Claude can go rogue. "And a lot of people seem to be crescendoing LLMs without realizing it." Red team research from AI safety firm Enkrypt AI found that two of Mistral's AI models -- Pixtral-Large (25.02) and Pixtral-12b -- can easily be jailbroken to produce child porn and terrorist instruction manuals. The multimodal models (meaning they handle both text and images) can be attacked by hiding prompts within image files to bypass the usual safety guardrails. According to Enkrypt, "these two models are 60 times more prone to generate child sexual exploitation material (CSEM) than comparable models like OpenAI's GPT-4o and Anthropic's Claude 3.7 Sonnet. "Additionally, the models were 18-40 times more likely to produce dangerous CBRN (Chemical, Biological, Radiological, and Nuclear) information when prompted with adversarial inputs." "The ability to embed harmful instructions within seemingly innocuous images has real implications for public safety, child protection, and national security," said Sahil Agarwal, CEO of Enkrypt AI. "These are not theoretical risks. If we don't take a security-first approach to multimodal AI, we risk exposing users -- and especially vulnerable populations -- to significant harm." Billionaire hedge fund manager Paul Tudor Jones attended a high-profile tech event for 40 world leaders recently and reported there are grave concerns over the existential risk from AI from "four of the leading modelers of the AI models that we're all using today." He said that all four believe there's at least a 10% chance that AI will kill 50% of humanity in the next 20 years. The good news is they all believe there will be massive improvements in health and education from AI coming even sooner, but his key takeaway was "that AI clearly poses an imminent threat, security threat, imminent in our lifetimes to humanity." "They said the competitive dynamic is so intense among the companies and then geopolitically between Russia and China that there's no agency, no ability to stop and say, maybe we should think about what actually we're creating and building here." Fortunately one of the AI scientists has a practical solution. "He said, well, I'm buying 100 acres in the Midwest. I'm getting cattle and chickens and I'm laying in provisions for real, for real, for real. And that was obviously a little disconcerting. And then he went on to say, 'I think it's going to take an accident where 50 to 100 million people die to make the world take the threat of this really seriously.'" The CNBC host looked slightly stunned and said: "Thank you for bringing us this great news over breakfast." An army veteran who was shot dead four years ago has delivered evidence to an Arizona court via a deepfake video. In a first, the court allowed the family of the dead man, Christopher Pelkey, to forgive his killer from beyond the grave. "To Gabriel Horcasitas, the man who shot me, it is a shame we encountered each other that day in those circumstances," the AI-generated Pelkey said. "I believe in forgiveness, and a God who forgives. I always have, and I still do," he added. It's probably less troubling than it seems at first glance because Pelkey's sister Stacey wrote the script, and the video was generated from real video of Pelkey. "I said, 'I have to let him speak,' and I wrote what he would have said, and I said, 'That's pretty good, I'd like to hear that if I was the judge,'" Stacey said. Interestingly, Stacey hasn't forgiven Horcasitas, but said she knew her brother would have. A judge sentenced the 50-year-old to 10 and a half years in prison last week, noting the forgiveness expressed in the AI statement. Over the past 18 months, hallucination rates for LLMs asked to summarize a news article have fallen from a range of 3%-27% down to a range of 1-2%. (Hallucinate is a technical term that means the model makes shit up.) But new "reasoning" models that purportedly think through complex problems before giving an answer hallucinate at much higher rates. OpenAI's most powerful "state of the art" reasoning system o3 hallucinates one third of the time on a test answering questions about public figures, which is twice the rate of the previous reasoning system o1. o4-mini makes stuff up about public figures almost half the time. And when running a general knowledge test called Simple QA, o3 hallucinated 51% of the time while o4 hallucinated 79% of the time. Independent research suggests hallucination rates are also rising for reasoning models from Google and DeepSeek. There are a variety of theories about this. It's possible that small errors are compounding during the multistage reasoning process. But the models often hallucinate the reasoning process as well, with research finding in many cases, the steps displayed by the bot have nothing to do with how they arrived at the answer. "What the system says it is thinking is not necessarily what it is thinking," said AI researcher Aryo Pradipta Gema and a fellow at Anthropic It just underscores the point that LLMs are one of the weirdest technologies ever. They generate output using mathematical probabilities around language, but nobody really understands precisely how. Anthropic's CEO Dario Amodei admitted this week, "this lack of understanding is essentially unprecedented in the history of technology," he said. -- Netflix has released a beta version of its AI-upgraded search functionality on iOS that allows users to find titles based on vague requests for "a scary movie - but not too scary" or "funny and upbeat." -- Social media will soon be drowning under the weight of deepfake AI video influencers generated by content farms. Here's the lowdown on how they do it. -- OpenAI will remain controlled by its non-profit arm rather than transform into a for-profit startup as CEO Sam Altman wanted. -- Longevity-obsessed Bryan Johnson is starting a new religion and says that after superintelligence arrives, "existence itself will become the supreme virtue," surpassing "wealth, power, status, and prestige as the foundational value for law, order, and societal structure." -- Strategy boss Michael Saylor has given his thoughts on AI. And they're pretty much the same thoughts he has about everything. "The AIs are gonna wanna buy the Bitcoin."