Elon Musk Claims AI Training Has Exhausted Human Knowledge, Advocates for Synthetic Data

Elon Musk says the world is running out of data for AI training

Tesla/X CEO Elon Musk seems to believe that training AI models with solely human-made data is becoming impossible. Musk claims that there's a growing lack of real-world data with which to train AI models, including his Grok AI chatbot. "We've now exhausted basically the cumulative sum of human knowledge ... in AI training," Musk said during an X live-stream interview conducted by Stagwell chairman Mark Penn. "That happened basically last year." Recommended Videos Musk's comments reflect those of former OpenAI researcher Ilya Sutskever, who predicted last December that the AI industry had reached "peak data." Musk's solution to this issue -- synthetic data -- also mirrors the larger industry. Google, OpenAI, Anthropic, and Meta already leverage synthetic data to train their models. "The only way to supplement [real-world data] is with synthetic data, where the AI creates [training data]," Musk said. "With synthetic data ... [AI] will sort of grade itself and go through this process of self-learning." While the use of synthetic data can offer significant cost savings to companies, some studies suggest that over-reliance on synthetic data can lead to model collapse where the AI's responses become less creative and more biased over time as they're repeatedly trained on recursively generated data. The lack of human-derived data hasn't stopped X from spinning off its Grok AI feature into its own iOS app on Thursday. The chatbot and image generator, notable for their complete lack of intellectual property or content guardrails, used to only be available to folks shelling out $8 a month for an X premium account. However, the new app is free for anyone to download.

Elon Musk agrees that we've exhausted AI training data

Elon Musk concurs with other AI experts that there's little real-world data left to train AI models on. "We've now exhausted basically the cumulative sum of human knowledge .... in AI training," Musk said during a live-streamed conversation with Stagwell chairman Mark Penn streamed on X late Wednesday. "That happened basically last year." Musk, who owns AI company xAI, echoed themes former OpenAI chief scientist Ilya Sutskever touched on at NeurIPS, the machine learning conference, during an address in December. Sutskever, who said the AI industry had reached what he called "peak data," predicted a lack of training data will force a shift away from the way models are developed today. Indeed, Musk suggested that synthetic data -- data generated by AI models themselves -- is the path forward. "With synthetic data ... [AI] will sort of grade itself and go through this process of self-learning," he said. Other companies, including tech giants like Microsoft, Meta, OpenAI, and Anthropic, are already using synthetic data to train flagship AI models. Gartner estimates 60% of the data used for AI and an­a­lyt­ics projects in 2024 were syn­thet­i­cally gen­er­ated. Microsoft's Phi-4, which was open-sourced early Wednesday, was trained on synthetic data alongside real-world data. So were Google's Gemma models. Anthropic used some synthetic data to develop one of its most performant systems, Claude 3.5 Sonnet. And Meta fine-tuned its most recent Llama series of models using AI-generated data. Training on synthetic data has other advantages, like cost savings. AI startup Writer claims its Palmyra X 004 model, which was developed using almost entirely synthetic sources, cost just $700,000 to develop -- compared to estimates of $4.6 million for a comparably-sized OpenAI model. But there as disadvantages as well. Some research suggests that synthetic data can lead to model collapse, where a model becomes less "creative" -- and more biased -- in its outputs, eventually seriously compromising its functionality. Because models create synthetic data, if the data used to train these models has biases and limitations, their outputs will be similarly tainted.

Elon Musk agrees that we've exhausted the internet of AI training data | TechCrunch

Elon Musk concurs with other AI experts that there's little real-world data left to train AI models on. "We've now exhausted basically the cumulative sum of human knowledge .... in AI training," Musk said during a live-streamed conversation with Stagwell chairman Mark Penn streamed on X late Wednesday. "That happened basically last year." Musk, who owns AI company xAI, echoed themes former OpenAI chief scientist Ilya Sutskever touched on at NeurIPS, the machine learning conference, during an address in December. Sutskever, who said the AI industry had reached what he called "peak data," predicted a lack of training data will force a shift away from the way models are trained today. Indeed, Musk suggested that synthetic data -- data generated by AI models themselves -- is the path forward. "With synthetic data ... [AI] will sort of grade itself and go through this process of self-learning with synthetic data," he said. Other companies, including tech giants like Microsoft, Meta, OpenAI, and Anthropic, are already using synthetic data to train flagship AI models. Gartner estimates 60% of the data used for AI and an­a­lyt­ics projects in 2024 were syn­thet­i­cally gen­er­ated. Microsoft's Phi-4, which was open-sourced early Wednesday, was trained on synthetic data alongside real-world data. So were Google's Gemma models. Anthropic used some synthetic data to develop one of its most performant systems, Claude 3.5 Sonnet. And Meta fine-tuned its most recent Llama series of models using AI-generated data. Training on synthetic data has other advantages, like cost savings. AI startup Writer claims its Palmyra X 004 model, which was developed using almost entirely synthetic sources, cost just $700,000 to develop -- compared to estimates of $4.6 million for a comparably-sized OpenAI model. But there as disadvantages as well. Some research suggests that synthetic data can lead to model collapse, where a model becomes less "creative" -- and more biased -- in its outputs, eventually seriously compromising its functionality.

Have AI Companies Run Out of Training Data? Elon Musk Thinks So

Elon Musk and former OpenAI chief scientist Ilya Sutskever say that AI companies have run out of real-world data to train generative models on. "We've now exhausted basically the cumulative sum of human knowledge ... in AI training," Musk tells Stagwell chairman Mark Penn in an X livestream yesterday, per Tech Crunch. "That happened basically last year." Musk's comments came just a few days after Sutskever, who helped build ChatGPT, told the annual Neurips event that "we have achieved peak data and there'll be no more." If true, it means that all of the available data on the internet has already been used up to train AI models. PetaPixel reported on this phenomenon back in November when it came to light that OpenAI was struggling with its new model, Orion, which is allegedly not hitting internal expectations. Similarly, Google's newest iteration of Gemini is not much better than the previous one. While Anthropic has also delayed the release of its Claude model. One of the reasons cited is that "it's become increasingly difficult to find new, untapped sources of high-quality, human-made training data that can be used to build more advanced AI systems." Musk suggested that the way for AI companies to plug this gap is synthetic data, i.e. the content that generative AI models themselves produce. "The only way to supplement [real-world data] is with synthetic data, where the AI creates [training data]," Musk says. "With synthetic data ... [AI] will sort of grade itself and go through this process of self-learning." However, this method is not totally proven. One study suggested that AI models trained on AI images start churning out garbage images with the lead author comparing it to species inbreeding. "If a species inbreeds with their own offspring and doesn't diversify their gene pool, it can lead to a collapse of the species," says Hany Farid, a computer scientist at the University of California, Berkeley. Nevertheless, Tech Crunch notes that Microsoft, Meta, OpenAI, and Anthropic are all using synthetic data to train AI models with. While this method has obvious benefits such as cost-cutting, the model's functionality could be compromised because of inherent limitations within the training data.

Elon Musk says all human data for AI training 'exhausted'

Tech entrepreneur suggests move to self-learning synthetic data created by artificial intelligence models Artificial intelligence companies have run out of data for training their models and have "exhausted" the sum of human knowledge, Elon Musk has said. The world's richest person suggested technology firms would have to turn to "synthetic" data - or material created by AI models - to build and fine-tune new systems, a process already taking place with the fast-developing technology. "The cumulative sum of human knowledge has been exhausted in AI training. That happened basically last year," said Musk in an interview livestreamed on his social media platform, X. AI models such as the GPT-4o model powering the ChatGPT chatbot are "trained" on a vast array of data taken from the internet, where they in effect learn to spot patterns in that information - allowing them to predict, for instance, the next word in a sentence. Musk said the "only way" to counter the lack of source material for training new models was to move to synthetic data created by AI. Referring to the exhaustion of data troves, he said: "The only way to then supplement that is with synthetic data where ... it will sort of write an essay or come up with a thesis and then will grade itself and ... go through this process of self-learning." Meta, the owner of Facebook and Instagram, has used synthetic data to fine-tune its biggest Llama AI model, while Microsoft has also used AI-made content for its Phi-4 model. Google and OpenAI, the company behind ChatGPT, have also used synthetic data in their AI work. However, Musk also warned that AI models' habit of generating "hallucinations" - a term for inaccurate or nonsensical output - was a danger for the synthetic data process. He told the livestreamed interview with Mark Penn, the chair of the advertising group Stagwell, that hallucinations had made the process of using artificial material "challenging" because "how do you know if it ... hallucinated the answer or it's a real answer". High-quality data, and control over it, is one of the legal battlegrounds in the AI boom. OpenAI admitted last year it would be impossible to create tools such as ChatGPT without access to copyrighted material, while the creative industries and publishers are demanding compensation for use of their output in the model training process.