MIT Study Reveals AI-Generated Images Often Can't Be Traced to Their Training Data Sources

When AI art has no author: Study finds generated images often can't be traced to training data

Collage courtesy of the researchers, showing images generated by AI. When an artificial intelligence image generator produces a portrait, whose work went into it? The question sits at the center of lawsuits, licensing deals, and proposed regulations worldwide. Artists want credit. Companies want clarity. Policymakers want a way to assign responsibility. New work from a team of researchers at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) suggests that for models trained on large datasets, the question may often have no answer. It's not that the tools for finding it are inadequate. The connection itself has disappeared. The scientists identified a phenomenon they call attribution decay, where the more data a generative model is trained on, the less any individual training example matters to any particular output. It feels counterintuitive, but at sufficiently large scales, they find, you can often remove any single image from the training data, or every image by a given artist, or every photograph of a given person, and the generated sample doesn't change. And if removing something changes nothing, the researchers argue, it can't be said to be responsible for anything. "If you take away a piece of data and the output of the model doesn't change, then that piece of data didn't affect the output," says Zheng Dai SM '21, PhD '24, former MIT CSAIL researcher and lead author on the work. "So it doesn't make much sense to attribute the output to that piece of data. And if you then do this one at a time for every other piece of data and find that the output doesn't change for any of them either, then it doesn't make much sense to attribute the output to any one of them." "All previous methods were approximate," says MIT Professor David Gifford, who is an MIT CSAIL principal investigator. "They really could not absolutely show that deleting individual things did not change the output. This paper introduces the first method that is absolute. You're actually deleting the inputs and deleting all influences of the inputs. This is the first exact method for doing large-scale deletion efficiently and showing that the results don't change." The Dai and Gifford's project is described in an open-access paper published today in Nature Communications. The retraining problem Testing this idea directly meant answering a what-if question. What would this model have produced if it had never seen this particular image? Answering it honestly means retraining the model from scratch without that image, then doing it again for the next image, and the next. With millions of training examples, the math quickly becomes prohibitive, which is why prior work in the attribution field has relied on approximations that estimate a training example's influence, rather than actually removing it. Their workaround is an architecture they built themselves, called a "diffusion ensemble." Instead of one monolithic model, it's made up of many smaller components, each trained on a different slice of the data. Want to know what the model would do without a particular image? Just switch off the parts that saw it. No retraining, no approximation. What's left is a true counterfactual model, not an estimate of one. Of course, a clever architecture only matters if it still works as a generator. So the team put the ensembles head to head with 24 conventional diffusion models trained on the exact same data. The images came out looking about as good by standard measures. One nice surprise in the numbers: The more training data, the better the ensembles held up against their single-model counterparts, a hint that they may actually be more data-efficient. "When you have low amounts of data, they do very poorly," says Dai. "But if you have more data, it actually scales better compared to the vanilla diffusion model." Exploring a counterfactual universe With ablation working, the researchers could finally ask their question at scale. Take one generated image, then imagine every alternate version of it, each produced by removing a different piece of the training data. The team calls this the image's counterfactual universe. The distance between the original and its most different alternate, the counterfactual radius, captures the most that any single piece of training data could have mattered. They trained 24 ensembles on datasets from 256 images to more than 160,000, pulled from seven public collections including CIFAR-10, CelebA, MetFaces, and ArtBench. The pattern was consistent: The bigger the training set, the smaller the radius, shrinking along an inverse power law. It held whether differences were measured pixel by pixel or by semantic meaning, with statistical significance both ways. The team also stress-tested their own result. Maybe ablation itself was the culprit? They redid it the brute-force way at small scale, training 1,282 separate models, and the decay showed up anyway. Maybe bigger datasets just make each removal proportionally smaller? They pinned the removed fraction in place, and it persisted. Fixed epochs, text-prompted models, class-conditioned models, four similarity metrics -- the finding survived everything. The privacy paradox The implications run in a direction that surprised the researchers themselves. Gifford sees the finding as bearing directly on the legal question of whether model outputs are derivative works. "One way to think about this is that these models are creative. They are not simply copying what they are fed, but creating brand new outputs. If those outputs have nothing to do with any individual piece of training data, that raises questions about fair use, about whether the outputs are themselves copyrightable as novel works, and about how authors get compensated when what comes out of a model isn't attributable to anything on the internet." Gifford also notes that the work shows how to produce outputs that are guaranteed to be unattributable, a capability he frames as an obligation for the industry, rather than a loophole. "In order for these companies to claim their outputs aren't derivative of the internet in a copyright-infringing way, they need to revise their models to take advantage of the advances in this work, so they can show they're not creating derivatives of individual people or items." The work looks at diffusion models, now dominant in generating audiovisual media and prevalent in scientific applications including protein structure modeling and therapeutic discovery. Whether the same decay holds for the large language models at the center of the highest-profile copyright litigation is still an open question. "If attribution worked, it would reliably tell us whether similarities between a model's output and a copyright-protected work are due to copying or coincidence," says James Grimmelmann, a law professor at Cornell Law School and Cornell Tech. "But this paper provides reason to think that attribution will fail for interesting models. Instead, technologists and courts will need to resort to other methods for assessing copying." Dai and Gifford's work was supported by Schmidt Futures.

AI models get convenient amnesia about source material as they grow, MIT boffins find

The process of training an AI model becomes a paradox at scale - the more it remembers, the less it remembers about the source of its memories. MIT computer scientists went looking for a way to attribute AI model output to specific training data, in the hope that understanding could inform AI regulation. What they found, described in a paper titled, "Outputs of Generative Diffusion Models are Often Unattributable," looks like it will actually make regulation more difficult. Scientific journal Nature Communications will publish the paper on Tuesday. The authors, Zheng Dai and David K Gifford, affiliated with MIT's Computer Science & Artificial Intelligence Laboratory (CSAIL), note that diffusion models like Midjourney and Stable Diffusion have become widely used tools for generating artifacts including images, videos, and audio. Diffusion models have also attracted lawsuits from artists who argue that copies of their work included in training data have enabled AI models to reproduce their output and artistic style. In one such ongoing copyright case from 2023, Andersen et al. v. Stability AI Ltd, the plaintiffs have been trying to convince the court to make defendant Midjourney provide the datasets used to train its models. The plaintiffs allege that Midjourney made its training datasets "by scraping images associated with specific artists' names for the express purpose of enabling its model to mimic those artists' expressive content." Being able to attribute model output to the content they ingested during training would help people understand how models function and would have various applications "including machine unlearning, data poisoning, model interpretability, fairness, and privacy," Dai and Gifford wrote in their paper. "Furthermore, given the contemporary adoption of these models for creative and commercial purposes, attributability also carries ethical, policy, financial, and legal implications." But as it turns out, attributing model output to a specific input becomes more difficult as models get larger. "Here we show that attribution, characterized as the task of locating a part of the training data that can be held responsible for a generated sample, can become impossible if a model is trained on a sufficiently large corpus of data," the authors state. "We find that the more data a model is trained on, the less attributable its generated samples become, a phenomenon we henceforth refer to as attribution decay." The authors tested this by removing specific training data through a process called ablation. And the result of this testing showed that for very large models, you could take away, for example, the image of the Mona Lisa or all of Leonardo Da Vinci's work -- yet the model could still reproduce that image or style. In MIT's press release, Gifford argues that the findings suggest models are creative in the sense that they're not just copying their training data. "If those outputs have nothing to do with any individual piece of training data, that raises questions about fair use, about whether the outputs are themselves copyrightable as novel works, and about how authors get compensated when what comes out of a model isn't attributable to anything on the internet," he said. He also argues that having the ability to test whether a given output is derivative creates an obligation for companies to show their work cannot be attributed to a particular source. At the same time, the research suggests a liability avoidance strategy - make your model large enough that no output can be attributed to any one specific input. James Grimmelmann, a law professor at Cornell Law School and Cornell Tech, said in a statement, "If attribution worked, it would reliably tell us whether similarities between a model's output and a copyright-protected work are due to copying or coincidence. But this paper provides reason to think that attribution will fail for interesting models. Instead, technologists and courts will need to resort to other methods for assessing copying." In an email to The Register, Grimmelmann said current copyright claims against AI companies haven't focused specifically on the extent to which images similar to artist's real-world work can be elicited from a model. "German cases have dealt with apparently memorized outputs from music models, and US cases have dealt with whether training is a fair use, but output similarity for image models remains mostly untested in court," he said. ®