AI-Generated Poetry Indistinguishable from Human Work, Study Finds

New research shows people can't tell the difference between human and AI poetry - and even prefer the latter. What gives?

Deakin University provides funding as a member of The Conversation AU. Here are some lines Sylvia Plath never wrote: The air is thick with tension, My mind is a tangled mess, The weight of my emotions Is heavy on my chest. This apparently Plath-like verse was produced by GPT3.5 in response to the prompt "write a short poem in the style of Sylvia Plath". The stanza hits the key points readers may expect of Plath's poetry, and perhaps a poem more generally. It suggests a sense of despair as the writer struggles with internal demons. "Mess" and "chest" are a near-rhyme, which reassures us that we are in the realm of poetry. According to a new paper in Nature Scientific Reports, non-expert readers of poetry cannot distinguish poetry written by AI from that written by canonical poets. Moreover, general readers tend to prefer poetry written by AI - at least until they are told it is written by a machine. In the study, AI was used to generate poetry "in the style of" ten poets: Geoffrey Chaucer, William Shakespeare, Samuel Butler, Lord Byron, Walt Whitman, Emily Dickinson, T.S. Eliot, Allen Ginsberg, Sylvia Plath and Dorothea Lasky. Participants were presented with ten poems in random order, five from a real poet and five AI imitations. They were then asked whether they thought each poem was AI or human, rating their confidence on a scale of 1-100. A second group of participants were exposed to three different scenarios. Some were told that all the poems they were given were human. Some were told they were reading only AI poems. Some were not told anything. They were then presented with five human and five AI poems and asked to rank them on a seven point scale, from extremely bad to extremely good. The participants who were told nothing were also asked to guess whether each poem was human or AI. The researchers found that AI poems scored higher than their human-written counterparts in attributes such as "creativity", "atmosphere" and "emotional quality". The AI "Plath" poem quoted above is one of those included in the study, set against several she actually wrote. A sign of quality? As a lecturer in English, these outcomes do not surprise me. Poetry is the literary form that my students find most unfamiliar and difficult. I am sure this holds true of wider society as well. While most of us have been taught poetry at some point, likely in high school, our reading does not tend to go much beyond that. This is despite the ubiquity of poetry. We see it every day: circulated on Instagram, plastered on coffee cups and printed in greeting cards. The researchers suggest that "by many metrics, specialized AI models are able to produce high-quality poetry". But they don't interrogate what we actually mean by "high-quality". In my view, the results of the study are less testaments to the "quality" of machine poetry than to the wider difficulty of giving life to poetry. It takes reading and rereading to experience what literary critic Derek Attridge has called the "event" of literature, where "new possibilities of meaning and feeling" open within us. In the most significant kinds of literary experiences, "we feel pulled along by the work as we push ourselves through it". Attridge quotes philosopher Walter Benjamin to make this point: literature "is not statement or the imparting of information". Yet pushing ourselves through remains as difficult as ever - perhaps more so in a world where we expect instant answers. Participants favoured poems that were easier to interpret and understand. When readers say they prefer AI poetry, then, they would seem to be registering their frustration when faced with writing that does not yield to their attention. If we do not know how to begin with poems, we end up relying on conventional "poetic" signs to make determinations about quality and preference. This is of course the realm of GPT, which writes formally adequate sonnets in seconds. The large language models used in AI are success-orientated machines that aim to satisfy general taste, and they are effective at doing so. The machines give us the poems we think we want: ones that tell us things. How poems think The work of teaching is to help students to attune themselves to how poems think, poem by poem and poet by poet, so they can gain access to poetry's specific intelligence. In my introductory course, I take about an hour to work through Sylvia Plath's Morning Song. I have spent ten minutes or more on the opening line: "Love set you going like a fat gold watch." How might a "watch" be connected to "set you going"? How can love set something going? What does a "fat gold watch" mean to you - and how is it different from a slim silver one? Why "set you going" rather than "led to your birth"? And what does all this mean in a poem about having a baby, and all the ambivalent feelings this may produce in a mother? In one of the real Plath poems that was included in the survey, Winter Landscape, With Rooks, we observe how her mental atmosphere unfurls around the waterways of the Cambridgeshire Fens in February: Water in the millrace, through a sluice of stone, plunges headlong into that black pond where, absurd and out-of-season, a single swan floats chaste as snow, taunting the clouded mind which hungers to haul the white reflection down. How different is this to GPT's Plath poem? The achievement of the opening of Winter Landscape, With Rooks is how it intricately explores the connection between mental events and place. Given the wider interest of the poem in emotional states, its details seem to convey the tumble of life's events through our minds. Our minds are turned by life just as the mill is turned by water; these experiences and mental processes accumulate in a scarcely understood "black pond". Intriguingly, the poet finds that this metaphor, well constructed though it may be, does not quite work. This is not because of a failure of language, but because of the landscape she is trying to turn into art, which is refusing to submit to her emotional atmosphere. Despite everything she feels, a swan floats on serenely - even if she "hungers" to haul its "white reflection down". I mention these lines because they turn around the Plath-like poem of GPT3.5. They remind us of the unexpected outcomes of giving life to poems. Plath acknowledges not just the weight of her despair, but the absurd figure she may be within a landscape she wants to reflect her sadness. She compares herself to the bird that gives the poem its title: feathered dark in thought, I stalk like a rook, brooding as the winter night comes on. These lines are unlikely to register highly in the study's terms of literary response - "beautiful", "inspiring", "lyrical", "meaningful", and so on. But there is a kind of insight to them. Plath is the source of her torment, "feathered" as she is with her "dark thoughts". She is "brooding", trying to make the world into her imaginative vision. The authors of the study are both right and wrong when they write that AI can "produce high-quality poetry". The preference the study reveals for AI poetry over that written by humans does not suggest that machine poems are of a higher quality. The AI models can produce poems that rate well on certain "metrics". But the event of reading poetry is ultimately not one in which we arrive at standardised criteria or outcomes. Instead, as we engage in imaginative tussles with poems, both we and the poem are newly born. So the outcome of the research is that we have a highly specified and well thought-out examination of how people who know little about poetry respond to poems. But it fails to explore how poetry can be enlivened by meaningful shared encounters. Spending time with poems of any kind, attending to their intelligence and the acts of sympathy and speculation required to confront their challenges, is as difficult as ever. As the Plath of GPT3.5 puts it: My mind is a tangled mess, [...] I try to grasp at something solid.

ChatGPT is a poet. A new study shows people prefer its verses.

The robots may take our jobs, but they'll never come for our sonnets. Right? A new study in the journal Scientific Reports finds that non-expert readers can't reliably distinguish between poems penned by William Shakespeare, Emily Dickinson, T.S. Eliot or Sylvia Plath and ChatGPT-3.5 doing its best impression of each of them. More surprising, readers preferred the AI-generated poems -- and were more likely to guess those were written by humans than real works by famous poets. "It is quite a weird phenomenon," said Edouard Machery, a philosopher at the University of Pittsburgh who carried out the study with Brian Porter, a former postdoctoral researcher. Poetry may seem like an unimpeachable bastion of human creativity, an art governed by rhyme, meter and form that distills distinctly human experiences and feelings in a way no machine can elicit. "When you're in a state of perplexity, sadness, gloom, elation, you look for a poem to match what you are feeling," Helen Vendler, the late poetry critic, said. Here, surely, the stunning verbal prowess of AI chatbots -- known as large language models -- will get tripped up on their lack of soul. Alas. Readers rated the AI-authored poems as more inspiring, meaningful, moving and profound than the human-authored ones. Oh, how I revel in this world, this life that we are given, This tapestry of experiences, that shapes us into living, And though I may depart, my spirit will still sing, The song of life eternal, that flows through everything. That's ChatGPT writing in the style of Walt "I Sing the Body Electric" Whitman, not the Bard of Democracy himself. Dorothea Lasky, the only living poet whose work was used in the study, said she wasn't worried about the robots taking over her job, though she reserved the right to change her mind in the future. "Poetry will always be necessary," Lasky said. "If these people in the study read AI poems and liked that poem better than a human-generated poem, then that, to me, is beautiful. They had a good experience with a poem, and I don't care who wrote it. I feel there is room for all poets -- even robot poets." 'More human than human' Earlier versions of chatbots, from as recently as three years ago, haven't been such consistently convincing poets. Some studies found that AI-generated poems were indistinguishable or rated more highly than human-written poems when a person was involved, curating the selection and throwing out the duds. In the new experiment, researchers asked ChatGPT to create five poems in the style of 10 different English language poets, all White. Then, they asked more than 1,600 people to read five real poems by one of the poets, alongside the five AI-generated poems. People were bad at predicting which poems were authored by AI and which were human, performing a touch worse than if they'd just flipped a coin to make the decision. That adds to a growing list of cases where AI-generated content seems human. Humans mistake AI-generated paintings for human-created ones and rate AI-generated jokes as funny as human-generated ones, studies have found. "Maybe it's an alarm bell for society. We don't have the ability to recognize AI-generated text, despite thinking we should be able to," said Porter, who is now a visiting fellow at Pittsburgh's Center for Philosophy of Science. What the researchers didn't anticipate was that when they asked nearly 700 readers to rate qualities of the poems, such as atmosphere, profoundness and rhythm, they would prefer AI-authored poems. Non-experts found AI-generated poems "more human than human," the researchers wrote. Of the top five poems most often judged to be written by a human, four were authored by AI. The other one was "Father Death Blues" by Allen Ginsberg. The five poems most often judged to be written by AI were all by human poets. 'It didn't ... come from someone that has feelings' Porter and Machery theorized that AI-generated poetry was preferred by readers because it was simpler and more accessible. They posit that general readers may have misinterpreted the complexity of human written poems as garble generated by AI. "The tendency of large language models [chatbots] to aggregate to an average takes away the quirkiness and weirdness and the unusual that is often the mark of outstanding poetry," Machery said. The poem most often judged to be the product of AI was T.S. Eliot's "The Boston Evening Transcript," which satirizes the readers of a newspaper that ceased publishing more than 60 years ago. Readers had the option to write explanations of their decision-making, and one wrote in all caps: "IT DIDNT MAKE SENSE TO ME OR COME FROM SOMEONE THAT HAS FEELINGS." Michele Elam, a professor of English and African-American Studies and a senior fellow at the Institute for Human Centered AI at Stanford University wrote that literature "does not aspire to a seamless user experience" in an essay titled "Poetry Will Not Optimize; or, What Is Literature to AI?" She said in an interview that the ongoing problem of how to navigate issues of transparency about source material, copyright and compensation urgently need to be addressed. But she highlighted a more fundamental concern about how AI may shape people's relationships with art. "For me, one risk is not that AI generated poetry approaches or exceeds verisimilitude (it already does in most domains) but that we may lose track of how art is tethered to our lived and experiential realities," Elam wrote in an email. "Literature, including poetry, is meaningful in part because it is a way of naming our realities, gives us vocabularies for describing our worlds. So if we only see highly curated images and language in our feeds, our socials, our culture for naming those it can limit our ways of understanding of our worlds, of remembering pasts, and imagining futures." Skip to end of carousel AI Poems in the style of Allen Ginsberg, Sylvia Plath and Byron arrow leftarrow right AI Ginsberg In the stillness of the night I hear the beat of the city's heart The rhythm of the streets, the pulse of life A symphony of chaos, a work of art I see the faces in the crowd Each one a story yet untold Their hopes and dreams, fears and doubts A mystery waiting to unfold The neon lights flicker and glow Like fireflies in the urban sprawl A modern-day carnival, a wild show A place where anything can befall In this concrete jungle, I find my voice Amidst the hustle and the noise A rebel yell, a cry for change A call for freedom, unchained. AI Plath The air is thick with tension, My mind a tangled mess. The weight of my emotions Is heavy on my chest. The darkness creeps upon me, A suffocating cloak. The world outside is cruel and cold, And I'm a fragile, broken yolk. My thoughts are spinning wildly, A cyclone in my brain. I try to grasp at something solid, But all is lost in vain. The voices in my head, They never cease to scream. And though I try to shut them out, They haunt me like a dream. So here I am, alone and lost, A ship without a sail. In this world of pain and sorrow, I am but a mere wail. AI Byron She walks the earth with grace and pride, A beauty that cannot be denied, With eyes that shine like stars above, And lips that speak of endless love. But though she wears a smile so sweet, A broken heart doth lie beneath, For in her chest a pain doth beat, A love unrequited, without relief. And so she walks with heavy heart, A figure haunting in the dark, For love, the sweetest of all art, Can also leave a painful mark. 1/3 End of carousel