AI creativity matches average humans but top creators still outperform machines, new studies show

As AI-Generated Music Advances, Humans Still Lead in Creativity, CMU Research Finds

AI can write songs, but still has a way to go before matching the creativity of tunes made by people, according to Carnegie Mellon University research. An interdisciplinary team examined what's missing when algorithms replace human experience, finding that AI-assisted music was slower, used fewer notes and was judged by listeners as less creative. Generative AI tools use large language models (LLMs), systems that learn from text and can create things -- like stories, answers, or even music -- based on user instructions. "We're trying to understand how these tools shape music and if they can support creative ideation when composing songs," said Jose Oros, who is working toward a doctorate degree in information systems from the Heinz College of Information Systems and Public Policy(opens in new window). Oros worked with Rahul Telang(opens in new window), Trustees Professor of Information Systems at Heinz College, and Richard Randall(opens in new window), associate professor of music theory in the School of Music(opens in new window) in the College of Fine Arts(opens in new window). Oros initiated a study where 140 musically trained participants created a 15-second melody with a small piano keyboard. Randomly selected participants were given access to a generative-AI platform called Udio, which they could use to generate tunes from text prompts for inspiration, while others wrote their own melodies without AI assistance. Then, the melodies were all judged by another group based on creativity, enjoyment and musicality. "A lot of studies on the effect of AI focus on productivity, but creativity and novelty are central outcomes that we care about in music and the arts in general," Oros said. "These tools are being developed with the promise of improving creativity or having a social benefit, so if these tools are not helping, then that has important implications. If these tools are helping, then we may want to enable their development."

AI and humans collide in world's biggest creativity experiment

Your LLM is more creative than some people - but that's not the whole story AI has passed a new benchmark, scoring better than the average human on a recognized creativity test involving 100,000 people. But there's more to the story than the results, underpinning how difficult it is to put "creativity" in a measurable box. Université de Montréal researchers led a large-scale study that pit 100,000 humans against the current leading generative AI models in an attempt to assess the creative power of both. It's the largest comparative study of its kind ever conducted. In order to measure what most of us would instantly consider a subjective field, the team used divergent linguistic creativity tasks to score the latest LLMs including ChatGPT, Claude and Gemini, as well as the humans. "We developed a rigorous framework that allows us to compare human and AI creativity using the same tools, based on data from more than 100,000 participants," said Professor Karim Jerbi, from the Department of Psychology at the Université de Montréal. The first caveat should be made here: It's obviously very hard to quantify human creativity in a way that can be compared with a LLM. So while this is a massive study, it's still defined by the measures and constraints that the scientists employed. The team used the Divergent Association Task (DAT), something used in psychology to measure a specific type of creativity. Essentially, it asks someone to come up with 10 words in four minutes, and the less related the words are, the more creative the list is considered to be. Then the scientists had the AI models do the same. What they found was that while LLMs demonstrated more creativity - as measured by the DAT - than a large number of humans, around half of the participants fared better than AI, and the top 10% far exceeded the performances of their computer challengers. So yes, while some people failed to show more "divergent creativity" than Claude, for example, a whole lot of people didn't. And this pulls into sharp focus just how difficult it is for even today's most advanced machines to replicate the output of the human brain - even after their creators have scraped what feels like every word in every language on Earth. "Even though AI can now reach human-level creativity on certain tests, we need to move beyond this misleading sense of competition," said Jerbi. "Generative AI has above all become an extremely powerful tool in the service of human creativity: It will not replace creators, but profoundly transform how they imagine, explore, and create - for those who choose to use it." So while LLMs are better than some humans when it comes to specific creative tasks, the same can be said when assessing a group of people. And this study highlights how complex and nuanced measuring human traits are - and how LLM benchmark scores aren't really solid indicators to use in comparative analyses. "Even though AI can now reach human-level creativity on certain tests, we need to move beyond this misleading sense of competition," said Jerbi. "Generative AI has above all become an extremely powerful tool in the service of human creativity: It will not replace creators, but profoundly transform how they imagine, explore, and create - for those who choose to use it." The researchers also investigated how AI models compared with humans when it came to creative writing tasks, including haikus, film synopses and short stories. Once again, the most creative humans outperformed the machines - even if LLMs overall scored better than the average participant. And it's worth noting that the LLMs expressed the most creativity when they were guided well - by humans. So it appears we are still a long way off from being replaced. And while AI has infiltrated our daily lives, there's a growing pushback on AI slop and using technology that exploits artists. Recently, some 800 artists have banded together to campaign against the use of AI-generated content in a broad range of creative fields. In this study, the researchers note that rather than think of it as a "human versus machine" investigation, the work should instead highlight AI's ability to assist people in creative endeavors. "By directly confronting human and machine capabilities, studies like ours push us to rethink what we mean by creativity," added Jerbi.