Meta's Aggressive Push to Surpass OpenAI's GPT-4 with Llama 3: Revelations from Court Filings

Meta execs obsessed over beating OpenAI's GPT-4 internally, court filings reveal | TechCrunch

Executives and researchers leading Meta's AI efforts obsessed over beating OpenAI's GPT-4 model while developing Llama 3, according to internal messages unsealed by a court on Tuesday in one of the company's ongoing AI copyright cases, Kadrey v. Meta. "Honestly... Our goal needs to be GPT-4," said Meta's VP of Generative AI, Ahmad Al-Dahle, in an October 2023 message to Meta researcher Hugo Touvron. "We have 64k GPUs coming! We need to learn how to build frontier and win this race." Though Meta releases open AI models, the company's AI leaders were far more focused on beating competitors that don't typically release their model's weights, like Anthropic and OpenAI, and instead gate them behind an API. Meta's execs and researchers held up Anthropic's Claude and OpenAI's GPT-4 as a gold standard to work toward. The French AI startup Mistral, one of the biggest open competitors to Meta, was mentioned several times in the internal messages, but the tone was dismissive. "Mistral is peanuts for us," Al-Dahle said in a message. "We should be able to do better," he said later. Tech companies are racing to upstage each other with cutting-edge AI models these days, but these court filings reveal just how competitive Meta's AI leaders truly were - and seemingly still are. At several points in the message exchanges, Meta's AI leads talked about how they were "very aggressive" in obtaining the right data to train Llama; at one point, an exec even said that "Llama 3 is literally all I care about," in a message to coworkers. Prosecutors in this case allege that Meta's executives occasionally cut corners in their mad race to shipping AI models, training on copyrighted books in the process. Touvron noted in a message that the mix of datasets used for Llama 2 "was bad," and talked about how Meta could use a better mix of data sources to improve Llama 3. Touvron and Al-Dahle then talked about clearing the path to use the LibGen dataset, which contains copyrighted works from Cengage Learning, Macmillan Learning, McGraw Hill, and Pearson Education. "Do we have the right datasets in there[?]" said Al-Dahle. "Is there anything you wanted to use but couldn't for some stupid reason?" Meta CEO Mark Zuckerberg has previously said he's trying to close the performance gap between Llama's AI models and closed models from OpenAI, Google, and others. The internal messages reveal the intense pressure within the company to do so. "This year, Llama 3 is competitive with the most advanced models and leading in some areas," said Zuckerberg in a letter from July 2024. "Starting next year, we expect future Llama models to become the most advanced in the industry." When Meta ultimately released Llama 3 in April 2024, the open AI model was competitive with leading closed models from Google, OpenAI, and Anthropic, and outperformed open options from Mistral. However, the data Meta used to train its models -- data Zuckerberg reportedly gave the green light to use, despite its copyright status -- are facing scrutiny in several ongoing lawsuits.

Inside Meta's race to beat OpenAI: "We need to learn how to build frontier and win this race"

In an October 2023 email to Meta AI researcher Hugo Touvron, Ahmad Al-Dahle, Meta's vice president of generative AI, wrote that the company's goal "needs to be GPT4," referring to the large language model OpenAI announced in March of 2023. Meta had "to learn how to build frontier and win this race," Al-Dahle added. Those plans apparently involved the book piracy site Library Genesis (LibGen) to train its AI systems. An undated email from Meta director of product Sony Theakanath, sent to VP of AI research Joelle Pineau, weighed whether to use LibGen internally only, for benchmarks included in a blog post, or to create a model trained on the site. In the email, Theakanath writes that "GenAI has been approved to use LibGen for Llama3... with a number of agreed upon mitigations" after escalating it to "MZ" -- presumably Meta CEO Mark Zuckerberg. As noted in the email, Theakanath believed "Libgen is essential to meet SOTA [state-of-the-art] numbers," adding "it is known that OpenAI and Mistral are using the library for their models (through word of mouth)." Mistral and OpenAI haven't stated whether or not they use LibGen. (The Verge reached out to both for more information).