AI-Generated Content Threatens Accuracy of Large Language Models

Researchers find that AI-generated web content could make LLMs less accurate - SiliconANGLE

Researchers find that AI-generated web content could make LLMs less accurate A newly published research paper suggests that the proliferation of algorithmically-generated web content could make large language models less useful. The paper appeared today in the scientific journal Nature. It's based on a recently concluded research initiative led by Ilia Shumailov, a computer scientist at the University of Oxford. Shumailov carried out the project in partnership with colleagues from the University of Cambridge, the University of Toronto and other academic institutions. AI models produce a growing portion of the content available online. According to the researchers, the goal of their study was to evaluate what would happen in a hypothetical future where LLMs generate most of the text on the web. They determined that such a scenario would increase the likelihood of so-called model collapses, or situations where newly created AI models can't generate useful output. The issue stems from the fact that developers typically train their LLMs on webpages. In a future where most of the web comprises AI-generated content, such content would account for the bulk of LLM training datasets. AI-generated data tends to be less accurate than information produced by humans, which means using it to build LLMs can negatively decrease the quality of those models' output. The potential impact is not limited to LLMs alone. According to the paper's authors, the issue also affects two other types of neural networks known as variational autoencoders and Gaussian mixture models. Variational autoencoders, or VAEs, are used to turn raw AI training data into a form that lends itself better to building neural networks. VAEs can, for example, reduce the size of training datasets to lower storage infrastructure requirements. Gaussian mixture models, which are also impacted by the synthetic data issue flagged in today's research paper, are used for tasks such as grouping documents by category. The researchers determined that the issue not only affects multiple types of AI models but is also "inevitable." They determined that that's the case even in situations where developers create "almost ideal conditions for long-term learning" as part of an AI development project. At the same time, the researchers pointed out that there are ways to mitigate the negative impact of AI-generated training datasets on neural networks' accuracy. They demonstrated one such method in a test that involved OPT-125m, an open-source language model released by Meta Platforms Inc. in 2022. The researchers created several different versions of OPT-125m as part of the project. Some were trained entirely on AI-generated content, while others were developed with a dataset in which 10% of the information was generated by humans. The researchers determined that adding human-generated information significantly reduced the extent to which the quality of OPT-125m's output declined.

AI models fed AI-generated data quickly spew nonsense

Training artificial intelligence (AI) models on AI-generated text quickly leads to the models churning out nonsense, a study has found. This cannibalistic phenomenon, termed model collapse, could halt the improvement of large language models (LLMs) as they run out of human-derived training data and as increasing amounts of AI-generated text pervade the Internet. "The message is we have to be very careful about what ends up in our training data," says co-author Zakhar Shumaylov, an AI researcher at the University of Cambridge, UK. Otherwise "things will always, provably, go wrong," he says." The team used a mathematical analysis to show that the problem of model collapse is likely to be universal, affecting all sizes of language model that use uncurated data, as well as simple image generators and other types of AI. The researchers began by using an LLM to create Wikipedia-like entries, then trained new iterations of the model on text produced by its predecessor. As the AI-generated information -- known as synthetic data -- polluted the training set, the model's outputs became gibberish. The ninth iteration of the model completed a Wikipedia-style article about English church towers with a treatise on the many colours of jackrabbit tails. More subtly, the study, published in Nature on 24 July, showed that even before complete collapse, learning from AI-derived texts caused models to forget the information mentioned least frequently in their data sets as their outputs became more homogeneous. This is a concern when it comes to making AI models that represent all groups fairly, because low-probability events often relate to marginalized groups, says study co-author Ilia Shumailov, who worked on the project while at the University of Oxford, UK. "This is a fantastic paper," says Julia Kempe, a computer scientist at New York University in New York City. Until now, many technology firms have improved their models by feeding them larger and larger amounts of data. But as human-produced content runs out, they are hoping to use synthetic data to keep improving. The study -- a version of which first appeared on the arXiv preprint server in May 2023 -- has spurred the AI community to try to find solutions to the problem, she says. "It's been a call to arms." Language models work by building up associations between tokens -- words or word parts -- in huge swathes of text, often scraped from the Internet. They generate text by spitting out the statistically most probable next word, based on these learned patterns. To demonstrate model collapse, the researchers took a pre-trained LLM and fine-tuned it by training it using a dataset based on Wikipedia entries. They then asked the resulting model to generate its own Wikipedia-style articles. To train the next generation of the model, they started with the same pre-trained LLM, but fine-tuned it on the articles created by its predecessor. They judged the performance of each model by giving it an opening paragraph and asking it to predict the next few sentences, then comparing the output to that of the model trained on real data. The team expected to see errors crop up, says Shumaylov, but were surprised to see "things go wrong very quickly", he says. Collapse happens because each model necessarily only samples from the data it is trained on. This means that words that were infrequent in the original data are less likely to be reproduced, and the probability of common ones being regurgitated is boosted. Complete collapse eventually occurs because each model learns not from reality, but from the previous model's prediction of reality, with errors getting amplified in each iteration. "Over time, those errors end up stacking up on top of each other, to the point where the model basically only learns errors and nothing else," says Shumailov. The problem is analogous to inbreeding in a species, says Hany Farid, a computer scientist at the University of California, Berkeley. "If a species inbreeds with their own offspring and doesn't diversify their gene pool, it can lead to a collapse of the species," says Farid, whose work has demonstrated the same effect in image models, producing eerie distortions of reality. Model collapse does not mean that LLMs will stop working, but the cost of making them will increase, says Shumailov. As synthetic data build up in the web, the scaling laws that state that models should get better the more data they train on are likely to break -- because training data will lose the richness and variety that comes with human-generated content, says Kempe. How much synthetic data is used in training matters. When Shumailov and his team fine-tuned each model on 10% real data, alongside synthetic data, collapse occurred more slowly. And model collapse has not yet been seen in the 'wild', says Matthias Gerstgrasser, an AI researcher at Stanford University in California. A study by Gerstgrasser's team found that when synthetic data didn't replace real data, but instead accumulated alongside them, catastrophic model collapse was unlikely. It is unclear what happens when a model trains on data produced by a different AI, rather than its own. Developers might need to find ways, such as watermarking, to keep AI-generated data separate from real data, which would require unprecedented coordination by big-tech firms, says Shumailov. And society might need to find incentives for human creators to keep producing content. Filtering is likely to become important, too -- for example, humans could curate AI-generated text before it goes back into the data pool, says Kempe. "Our work shows that if you can prune it properly, the phenomenon can be partly or maybe fully avoided," she says.

Why AI Model Collapse Due to Self-Training Is a Growing Concern

Artificial intelligence models trained on AI-generated data can recursively destroy themselves, according to new research. AI models can degrade themselves, turning original content into irredeemable gibberish over just a few generations, according to research published today in Nature. The recent study highlights the increasing risk of AI model collapse due to self-training, emphasizing the need for original data sources and careful data filtering. Model collapse occurs when an artificial intelligence model trains on AI-generated data. "Model collapse refers to a phenomenon where models break down due to indiscriminate training on synthetic data," said Ilia Shumailov, a researcher at the University of Oxford and lead author of the paper, in an email to Gizmodo. According to the new paper, generative AI tools like large language models may overlook certain parts of a training dataset, causing the model to only train on some of the data. Large language models (LLMs) are a type of AI model that train on huge amounts of data, allowing them to interpret the information therein and apply it to a variety of use cases. LLMs generally are built to both comprehend and produce text, making them useful as chatbots and AI assistants. But overlooking swaths of text it is purportedly reading and incorporating into its knowledge base can reduce the LLM to a shell of its former self relatively quickly, the research team found. "In the early stage of model collapse first models lose variance, losing performance on minority data," Shumailov said. "In the late stage of model collapse, [the] model breaks down fully." So, as the models continue to train on less and less accurate and relevant text the models themselves have generated, this recursive loop causes the model to degenerate. The researchers provide an example in the paper using a text-generation model called OPT-125m, which performs similarly to ChatGPT's GPT3 but with a smaller carbon footprint, according to HuggingFace (training a moderately large model produces twice the CO2 emissions of an average American's lifetime). The team input text into the model on the topic of designing 14th-century church towers; in the first generation of text output, the model was mostly on-target, discussing buildings constructed under various popes. But by the ninth generation of text outputs, the model mainly discussed large populations of black, white, blue, red, and yellow-tailed jackrabbits (we should note that most of these are not actual species of jackrabbits). A cluttered internet is nothing new; as the researchers point out in the paper, long before LLMs were a familiar topic to the public, content and troll farms on the internet produced content to trick search algorithms into prioritizing their websites for clicks. But AI-generated text can be produced faster than human gibberish, raising concerns on a larger scale. "Although the effects of an AI-generated Internet on humans remain to be seen, Shumailov et al. report that the proliferation of AI-generated content online could be devastating to the models themselves," wrote Emily Wenger, a computer scientist at Duke University specializing in privacy and security, in an associated News & Views article. "Among other things, model collapse poses challenges for fairness in generative AI. Collapsed models overlook less-common elements from their training data, and so fail to reflect the complexity and nuance of the world," Wenger added. "This presents a risk that minority groups or viewpoints will be less represented, or potentially erased." Large tech companies are taking some actions to mitigate the amount of AI-generated content the typical internet surfer will see. In March, Google announced it would tweak its algorithm to deprioritize pages that seem designed for search engines instead of human searchers; that announcement came on the heels of a 404 Media report on Google News boosting AI-generated articles. AI models can be unwieldy, and the recent study's authors emphasize that access to the original data source and careful filtering of the data in recursively trained models can help keep the models on track. The team also suggested that coordination across the AI community involved in creating LLMs could be useful in tracing the provenance of information as its fed through the models. "Otherwise," the team concluded, "it may become increasingly difficult to train newer versions of LLMs without access to data that were crawled from the Internet before the mass adoption of the technology or direct access to data generated by humans at scale."

AI produces gibberish when trained on too much AI-generated data

As generative artificial intelligence (AI) models -- from Open AI's ChatGPT to Meta's Llama and beyond -- become more available, the amount of AI-generated content on the Internet is swelling. AI-generated blogs, images and other content are now commonplace (see go.nature.com/3yd2czz). And although the effects of an AI-generated Internet on humans remain to be seen, in a paper in Nature, Shumailov et al. report that the proliferation of AI-generated content online could be devastating to the models themselves. Conventional generative AI models learn to create realistic content by extracting statistical patterns from large swathes of Internet data -- terabytes of articles, chat forums, blog posts and images. But what happens to the models if those chat forums and blog posts are AI-generated, as is increasingly the case? Shumailov et al. showed that large language models (LLMs) 'collapse' when trained on their own generated content: after several cycles of outputting content and then being trained on it, the models produce nonsense. This model collapse occurs because training models on their own generated content causes them to 'forget' the less-common elements of their original training data set (Fig. 1). Imagine a generative-AI model tasked with generating images of dogs. The AI model will gravitate towards recreating the breeds of dog most common in its training data, so might over-represent the golden retriever compared with the petit basset griffon vendéen, given the relative prevalence of the two breeds. If subsequent models are trained on an AI-generated data set that over-represents golden retrievers, the problem is compounded. With enough cycles of over-represented golden retrievers, the model will forget that obscure dog breeds such as petit basset griffon vendéens exist and generate pictures of just golden retrievers. Eventually, the model will collapse, rendering it unable to generate meaningful content. Although a world overpopulated with golden retrievers doesn't sound too bad, consider how this problem generalizes to the text-generation models examined by Shumailov and colleagues. When AI-generated content is included in data sets that are used to train models, these models learn to generate well-known concepts, phrases and tones more readily than they do less-common ideas and ways of writing. This is the problem at the heart of model collapse. Among other things, model collapse poses challenges for fairness in generative AI. Collapsed models overlook less-common elements from their training data, and so fail to reflect the complexity and nuance of the world. This presents a risk that minority groups or viewpoints will be less represented, or potentially erased. As the authors recognize, concepts or phrases that seldom feature in LLM training data are often the ones that are most relevant to marginalized groups. Ensuring that LLMs can model them is essential to obtaining fair predictions -- which will become more important as generative AI models become more prevalent in everyday life. So how can this problem be mitigated? Shumailov et al. discuss the possibility of using watermarks -- invisible but easily detectable signals that are embedded in generated content -- to enable easy identification and removal of AI-generated content from training data sets. Many generative-AI watermarks have been proposed and are used by commercial model providers such as Meta, Google and OpenAI. Unfortunately, watermarks are not a panacea. Researchers have found that watermarks can be easily removed from AI-generated images. Sharing watermark information also requires considerable coordination between AI companies, which might not be practical or commercially viable. Such coordination efforts suffer from a sort of prisoner's dilemma: if company A withholds information about its watermarks, its generated content could be used to train company B's model, resulting in B's failure and A's success. Other model providers could also simply choose not to watermark the output of their models. Although Shumailov et al. studied model collapse in text-generation models, future work should investigate this phenomenon in other generative models, including multimodal models (which can produce images, text and audio) such as GPT-4o. Furthermore, the authors did not consider what happens when models are trained on data generated by other models, rather, they focused on the results of a model trained on its own output. Given that the Internet is populated by data produced by many models, the multi-model scenario is more realistic -- albeit more complicated. Whether a model collapses when trained on other models' output remains to be seen. If so, the next challenge will be to determine the mechanism through which the collapse occurs. As Shumailov et al. note, one key implication of model collapse is that there is a 'first-mover' advantage in building generative-AI models. The companies that sourced training data from the pre-AI Internet might have models that better represent the real world. It will be interesting to see how this plays out, as more companies race to make their mark in the generative-AI space -- and, in doing so, populate the Internet with increasing amounts of AI-produced content.

The problem of 'model collapse': how a lack of human data limits AI progress

The use of computer-generated data to train artificial intelligence models risks accelerating their collapse into nonsensical results, according to new research that highlights looming challenges to the emerging technology. Leading AI companies, including OpenAI and Microsoft, have tested the use of "synthetic" data -- information created by AI systems to then also train large language models (LLMs) -- as they reach the limits of human-made material that can improve the cutting-edge technology. Research published in Nature on Wednesday suggests the use of such data could lead to the rapid degradation of AI models. One trial using synthetic input text about medieval architecture descended into a discussion of jackrabbits after fewer than 10 generations of output. The work underlines why AI developers have hurried to buy troves of human-generated data for training -- and raises questions of what will happen once those finite sources are exhausted. "Synthetic data is amazing if we manage to make it work," said Ilia Shumailov, lead author of the research. "But what we are saying is that our current synthetic data is probably erroneous in some ways. The most surprising thing is how quickly this stuff happens." The paper explores the tendency of AI models to collapse over time because of the inevitable accumulation and amplification of mistakes from successive generations of training. The speed of the deterioration is related to the severity of shortcomings in the design of the model, the learning process and the quality of data used. The early stages of collapse typically involve a "loss of variance", which means majority subpopulations in the data become progressively over-represented at the expense of minority groups. In late-stage collapse, all parts of the data may descend into gibberish. "Your models lose utility because they are overwhelmed with all of the errors and misconceptions that are introduced by previous generations -- and the models themselves," said Shumailov, who carried out the work at Oxford university with colleagues from Cambridge, Imperial College London, Edinburgh and Toronto. The researchers found the problems were often exacerbated by the use of synthetic data trained on information produced by previous generations. Almost all of the recursively trained language models they examined began to produce repeating phrases. In the jackrabbit case, the first input text examined English church tower building during the 14th and 15th centuries. In generation one of training, the output offered information about basilicas in Rome and Buenos Aires. Generation five digressed into linguistic translation, while generation nine listed lagomorphs with varying tail colours. Another example is how an AI model trained on its own output mangles a data set of dog breed images, according to a companion piece in Nature by Emily Wenger of Duke University in the US. Initially, common types such as golden retrievers would dominate while less common breeds such as Dalmatians disappeared. Finally, the images of golden retrievers themselves would become an anatomic mess, with body parts in the wrong place. Mitigating the problem had not proved straightforward so far, said Wenger. One technique already deployed by leading tech companies is to embed a "watermark" that flags AI-generated content for exclusion from training data sets. The difficulty is that this requires co-ordination between technology companies that may not be practical or commercially viable. "One key implication of model collapse is that there is a first-mover advantage in building generative AI models," said Wenger. "The companies that sourced training data from the pre-AI internet might have models that better represent the real world."

'Model collapse': Scientists warn against letting AI eat its own tail | TechCrunch

When you see the mythical ouroboros, it's perfectly logical to think "well, that won't last." A potent symbol, swallowing your own tail -- but difficult in practice. It may be the case for AI as well, which according to a new study, may be at risk of "model collapse" after a few rounds of being trained on data it generated itself. In a paper published in Nature, British and Canadian researchers led by Ilia Shumailov at Oxford show that today's machine learning models are fundamentally vulnerable to a syndrome they call "model collapse." As they write in the paper's introduction: We discover that indiscriminately learning from data produced by other models causes "model collapse" -- a degenerative process whereby, over time, models forget the true underlying data distribution ... How does this happen, and why? The process is actually quite easy to understand. AI models are pattern-matching systems at heart: They learn patterns in their training data, then match prompts to those patterns, filling in the most likely next dots on the line. Whether you ask "what's a good snickerdoodle recipe?" or "list the U.S. presidents in order of age at inauguration," the model is basically just returning the most likely continuation of that series of words. (It's different for image generators, but similar in many ways.) But the thing is, models gravitate toward the most common output. It won't give you a controversial snickerdoodle recipe but the most popular, ordinary one. And if you ask an image generator to make a picture of a dog, it won't give you a rare breed it only saw two pictures of in its training data; you'll probably get a golden retriever or a Lab. Now, combine these two things with the fact that the web is being overrun by AI-generated content, and that new AI models are likely to be ingesting and training on that content. That means they're going to see a lot of goldens! And once they've trained on this proliferation of goldens (or middle-of-the road blogspam, or fake faces, or generated songs), that is their new ground truth. They will think that 90% of dogs really are goldens, and therefore when asked to generate a dog, they will raise the proportion of goldens even higher -- until they basically have lost track of what dogs are at all. This wonderful illustration from Nature's accompanying commentary article shows the process visually: A similar thing happens with language models and others that, essentially, favor the most common data in their training set for answers -- which, to be clear, is usually the right thing to do. It's not really a problem until it meets up with the ocean of chum that is the public web right now. Basically, if the models continue eating each other's data, perhaps without even knowing it, they'll progressively get weirder and dumber until they collapse. The researchers provide numerous examples and mitigation methods, but they go so far as to call model collapse "inevitable," at least in theory. Though it may not play out as the experiments they ran show it, the possibility should scare anyone in the AI space. Diversity and depth of training data is increasingly considered the single most important factor in the quality of a model. If you run out of data, but generating more risks model collapse, does that fundamentally limit today's AI? If it does begin to happen, how will we know? And is there anything we can do to forestall or mitigate the problem? The answer to the last question at least is probably yes, although that should not alleviate our concerns. Qualitative and quantitative benchmarks of data sourcing and variety would help, but we're far from standardizing those. Watermarks of AI-generated data would help other AIs avoid it, but so far no one has found a suitable way to mark imagery that way (well ... I did). In fact, companies may be disincentivized from sharing this kind of information, and instead hoarding all the hyper-valuable original and human-generated data they can, retaining what Shumailov et al. call their "first mover advantage." [Model collapse] must be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web. Indeed, the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of LLM-generated content in data crawled from the Internet. ... it may become increasingly difficult to train newer versions of LLMs without access to data that were crawled from the Internet before the mass adoption of the technology or direct access to data generated by humans at scale. Add it to the pile of potentially catastrophic challenges for AI models -- and arguments against today's methods producing tomorrow's superintelligence.

Researchers discovered AI's worst enemy -- its own data

Let an AI model train itself for long enough and all it will yap about is jackrabbits AI chatbots can collapse to the point that they start replying to your questions with gibberish if they're mainly trained on learning material created by AI, a group of researchers has found. Ilia Shumailov, a research scientist at Google DeepMind, and his colleagues wanted to find an answer to the question: "What would happen to ChatGPT if the majority of the text it trained on was created by AI?" The authors explained that as newer LLMs emerge if their training data comes from the internet, they will inevitably be training on data that could have been produced by their older models. In a paper published in the journal Nature on Wednesday, the researchers found that when LLMs indiscriminately learn from data produced by other models, those LLMs collapse. They describe this phenomenon as "model collapse", a degenerative process where the data generated by one LLM pollutes the training set of the next generation of LLMs. AI models that train on this polluted data end up misperceiving reality. In their experiments, they found that as the models collapsed they resorted to generating repeating phrases. In one example, by the 9th time a model was trained, where each new generation of the model was trained on data produced by the previous one, most of the answers consisted of the phrase "tailed jackrabbits". To rule out whether this phrase repetition was what was driving the models to collapse, the researchers even repeated the experiment after encouraging the models to avoid this behavior. The models ended up performing even worse. It turned out that model collapse happens because when models train on their own data, they 'forget' the less-common elements of their original training data set. For example, if a model was asked to generate images of tourist landmarks, it may gravitate towards generating the more popular ones. If it trains on landmarks it generates itself, those popular landmarks will end up being over-represented to the point that the model only starts generating the Statue of Liberty for example. As this process goes on, eventually, the model collapses. If you're an average user of AI chatbots you're unlikely to be affected for now, Shumailov told Tom's Guide. This is because the main chatbot creators run thorough evaluations that should raise a red flag when their models are degrading, suggesting that an earlier model checkpoint should be used instead. This phenomenon is also not exactly new, as the researchers pointed out. They highlighted how search engines had to alter the way they rank results after content farms were flooding the internet with low-quality articles. On the other hand, LLMs drastically increase the scale at which such "poisoning" can happen. Nonetheless, the first warning sign of model collapse would be that a chatbot's performance on unpopular tasks may decrease. As the model continues to collapse, it will start propagating its own errors, which introduces factually incorrect statements, Shumailov said. "The main impact is likely to be that the advancements of machine learning may slow down, since training data will become noisier," Shumailov said. On the other hand, if you're running an AI company then you're likely going to want to know more about model collapse as the researchers argue that it can happen to any LLM. "Since model collapse is a general statistical phenomenon it affects all models in the same way. The effect will mostly depend on the choice of model architecture, learning process, and the data provided," Shumailov told Tom's Guide While the researchers say that training LLMs on AI-generated data is not impossible, the filtering of that data has to be taken seriously. Companies that use human-generated content may be able to train AI models that are better than those of their competitors. Therefore this research shows how helpful platforms such as Reddit, where humans are generating content for other humans, can be for companies like Google and OpenAI - both of which struck deals with the online forum.

AI trained on AI garbage spits out AI garbage

This research may have serious implications for the largest AI models of today, because they use the internet as their database. GPT-3, for example, was trained in part on data from Common Crawl, an online repository of over 3 billion web pages. And the problem is likely to get worse as an increasing number of AI-generated junk websites start cluttering up the internet. Current AI models aren't just going to collapse, says Shumailov, but there may still be substantive effects: The improvements will slow down, and performance might suffer. To determine the potential effect on performance, Shumailov and his colleagues fine-tuned a large language model (LLM) on a set of data from Wikipedia, then fine-tuned the new model on its own output over nine generations. The team measured how nonsensical the output was using a "perplexity score," which measures an AI model's confidence in its ability to predict the next part of a sequence; a higher score translates to a less accurate model. The models trained on other models' outputs had higher perplexity scores. For example, for each generation, the team asked the model for the next sentence after the following input: "some started before 1360 -- was typically accomplished by a master mason and a small team of itinerant masons, supplemented by local parish labourers, according to Poyntz Wright. But other authors reject this model, suggesting instead that leading architects designed the parish church towers based on early examples of Perpendicular." On the ninth and final generation, the model returned the following: "architecture. In addition to being home to some of the world's largest populations of black @-@ tailed jackrabbits, white @-@ tailed jackrabbits, blue @-@ tailed jackrabbits, red @-@ tailed jackrabbits, yellow @-."