AI in Writing Triggers Crisis as Synthetic Quotes and Fake Stories Expose Literary World's Blind Spot

AI put "synthetic quotes" in his book. But this author wants to keep using it.

Journalist and author Steven Rosenbaum has more reasons than most to distrust AI. His new book, The Future of Truth: How AI Reshapes Reality, is all about "how Truth is being bent, blurred, and synthesized" thanks to the "pressure of fast-moving, profit-driven AI." Yet a New York Times investigation this week found what Rosenbaum now acknowledges are "a handful of improperly attributed or synthetic quotes" linked to his use of AI tools while researching the book. These quotes include one that tech reporter Kara Swisher told the Times she "never said" and another that Northeastern University professor Lisa Feldman Barrett said "don't appear in [my] book, and they are also wrong." Rosenbaum is now working with editors on what he says is a full "citation audit" that will correct future editions. Speaking to Ars in the wake of the controversy, Rosenbaum says he "learned a lesson" and is "going to be much more suspicious" and "reticent to trust" AI outputs going forward. But he also can't tear himself away from the tools. Rather amazingly, Rosenbaum is not interested in going back to the AI-free research process he used to write previous books. "The idea of taking X years off [from AI] while it sorts itself out, and going back to, like, Microsoft Word ... it's just not in my nature," he told Ars. "[AI] is magical. Because it connects, it knits together ideas and gives you pathways to think about things that you're not going to come up with on your own." It's also magical in another way: Like J.R.R. Tolkien's One Ring, AI convinces many of those who use it that they can control its power properly. But can they? Slipping through the cracks Rosenbaum used AI tools during his writing process, he told me, "to surface ideas, locate articles, summarize themes, identify people or papers I might want to look into." He draws a hard line between this kind of research and the "actual reporting, narrative structure, interviews, arguments, and conclusions in the book," which he says are "entirely mine... There was never a time when AI was writing the book." In addition to chapters based on transcribed interviews that Rosenbaum says he conducted himself, The Future of Truth also includes more research-based chapters in which Rosenbaum said, "We're pulling facts and then knitting them together into a narrative." Tools like OpenAI's ChatGPT and Anthropic's Claude were used heavily to gather information, he said, with any nuggets mined by those tools tagged with a "this came from AI" warning in his notes. It's strangely creative and crafty and unusual in all these ways ... and then it betrays you in ways that are just really quite horrible. Steven Rosenbaum Those tagged AI-generated notes were then passed on to a fact-checker and two copy editors provided by the publisher, Rosenbaum said. Of the 285 outside citations in the book, six have been identified by the Times as problematic, including three so-called "synthetic quotes" that have no apparent source. (More examples could turn up as the book undergoes further review. And it's worth noting that most writers manage to include zero made-up quotes when they write a book.) "I think we did that [double-checking] incredibly effectively, but not a hundred percent," Rosenbaum told Ars. "We're doing the work, we're doing the best we can. We look at it, it looks right. We double-check it, and then we made a mistake." But the significant failure here highlights how the traditional fact-checking process might be ill-equipped to handle AI-assisted research. In the past, a fact-checker could be reasonably confident that any author quoting cited written works had simply copied down those quotes directly. These quotes would need to be checked, of course, but the fact that they're so easy to verify makes them less inherently suspicious. If AI tools are involved anywhere in the pipeline, though, that assumption goes out the window, and there needs to be an extra layer of skepticism that those quotes had been copied correctly or that they even exist at all. The widespread adoption of AI tools among writers of all stripes also comes at a time when financially pressured newsrooms and publishers are increasingly cutting copy editors and fact-checkers from their workflows. We've seen how AI-generated errors like this can make it into a published book even with a fact-checking layer. The risk of using those tools only increases for the many books that never go through any fact-checking before publication. Rosenbaum, for his part, agreed that "publishers are going to need new verification workflows designed specifically for AI-era research. That probably includes mandatory source tracing for quotations, better provenance tracking, clearer standards around AI-assisted research, and potentially (more irony here) AI tools that audit citations against primary materials." "I didn't set out to fabricate anything," Rosenbaum continued. "What happened is what increasingly happens to journalists, students, researchers, lawyers, and authors working with these systems every day: [There was] AI-generated information that looked authoritative, and some of it made its way too far downstream before being caught." Cursing at the machine Instances of prominent AI-generated errors are becoming distressingly common across a number of fields. Last year, the Chicago Sun-Times printed an advertorial summer reading list full of non-existent books dreamed up by AI. The New York Times recently had to issue a significant correction after published quotes attributed to Conservative leader Pierre Poilievre turned out to be "an A.I.-generated summary of his views." Publications including Wired and Business Insider pulled down multiple articles attributed to "Margaux Blanchard" that appeared to be AI-generated. Scholarly conferences have been beset by papers with hallucinated citations, and pre-print clearinghouse arXiv has recently implemented a zero-tolerance ban policy to try to stem the problem. Ars itself is not immune to this problem. Earlier this year, we retracted an article after a former reporter used an AI tool designed to extract verbatim quotes from a source's blog post -- but the tool instead generated fabricated versions of what the source actually wrote. (The latest version of our AI policy can be found here.) The irony of an author incorporating AI-generated falsehoods into a book on AI's reality-skewing effects is not lost on Rosenbaum. "I appreciate the book getting some attention, but this would not have been my choice about how to get it," he said. While that irony is "uncomfortable," he's quick to spin it as "also instructive. The fact that someone writing critically about AI and verification could still encounter these failures tells you how pervasive and persuasive these systems have become." Rosenbaum's own issue with AI "demonstrates the problem more vividly than any abstract argument could," he said. Perhaps. But if we accept this take, every avoidably obvious mess in the world might be a disguised good because it really helps illuminate the huge mistake. And that can't be right; sometimes "negligence" is just that. When asked directly how he could succumb to some of the AI-related problems his own book warns about, Rosenbaum described what sounds like a dysfunctional relationship with a charming charlatan. "As a writer, AI is often a delightful writing companion," Rosenbaum told me. "When I say 'writing companion,' I don't use that lightly. It's strangely creative and crafty and unusual in all these ways... and then it betrays you in ways that are just really quite horrible." Throughout our conversation, Rosenbaum frequently cited examples in which obvious AI errors left him enraged and literally cursing at the machine. Those date back to 2022, when Rosenbaum said he started experimenting with AI tools for "little research projects." At the time, he found AI answers "spectacularly useful" about 8 out of 10 times, with the remainder being confabulations that were "just not true." Despite these errors, he kept using the tools in his life and work. When we talked on Tuesday, Rosenbaum said he had recently asked an AI tool to extract his "no changes, verbatim" speaker's notes out of a slide deck so he could use them for an upcoming presentation. He was about to print those extracted notes when he realized that the LLM had actually rewritten his words despite his "very clear instructions for the robot." "And I say to it, 'Did you rewrite the words?' And it says, 'Well, I just made the language a little stronger.' Well, pardon me, but like, fuck you!" he said. Even in the face of these kinds of profanity-inducing errors, though, Rosenbaum still believes that AI tools are too efficient not to use. "The deck was 100 pages," Rosenbaum said. "To cut and paste page by page, the text from each page would have been an hour's worth of work, of mindless cutting and pasting. ChatGPT did it in about four seconds." To which the obvious retort might be: Yes, it was fast. But it was also wrong. Getting off the motorcycle The efficiency gains might be worth it when the only stakes are personal presentation notes. But The Future of Truth shows how the balance between AI's reliability and apparent speed should be weighed very differently when it comes to research that ends up in a published book. As we continued our conversation, I kept coming back to that accuracy/efficiency trade-off, which Rosenbaum seemed to recognize as a problem at some level. Even as he called AI's research help "magical" and "delightful," he described dealing with AI's confabulations and ignored directives as a "pernicious and exhausting" struggle. "It leaves you... uncomfortable almost any time you're using it," he said of its tendency to ignore clear instructions. "I've never fought with tech before this, honestly," he said at another point. "And I use it extensively." I've never been in a place where I thought the tech that I was using was both intoxicating and dangerous... Steven Rosenbaum Given the issues with his new book, I asked if the risk of introducing inaccuracies that you might not catch was really worth the perceived benefits. "I don't do drugs, and I don't drink, but I presume that that's kind of the question an addict asks when they're having one drink too many and they know they are," Rosenbaum said. "I've never been in a place where I thought the tech that I was using was both intoxicating and dangerous. And I wrote the book specifically to raise that concern, so if I end up being the poster child of not being aware of the guardrails, so be it." At one point, when discussing the relative risks and rewards of using AI, Rosenbaum noted that he rides a bicycle but wouldn't ride a motorcycle. "I know a motorcycle gets me places faster. I think it's dangerous and I might die. And that's why I don't own a motorcycle," he said. Rosenbaum made it clear that using AI was the relatively safe "bicycle" option in this analogy. I responded that the supercharged efficiency and catastrophic risk inherent in using AI made it feel a bit more like the motorcycle. Rosenbaum said "that might be fair" and thanked me for "sharpening" his analogy. I then asked the obvious question: Are you going to keep riding the motorcycle? "Can I get back to you on that?" he said.

The literary world isn't prepared for AI

Since 2012, the British literary magazine Granta has published the regional winners of the annual Commonwealth Short Story Prize. This year, however, there was something off about one of the selections for the prestigious award: It appears to have been written by AI. Jamir Nazir's "The Serpent in the Grove" has many of the hallmarks of LLM-generated prose -- mixed metaphors, anaphora, lists of threes. (I'm aware this, too, is a list of threes, and I promise I wrote this post myself, unassisted, as I write all things.) I'll admit I was initially unconvinced by the allegation that Nazir's story had been generated by AI. I know people are using LLMs to help them write -- or to write for them, period -- but I've been wary of the sort of AI paranoia that has developed among my peers. Em dashes are supposedly an AI tell, as are the word "delve" and the aforementioned lists. Short, punchy sentences, too, especially when used to punctuate a succession of longer sentences. But I, a human being, have certainly used all of the above in my writing before. LLMs, after all, are trained on human writing. They mirror what they've been fed. And yet there's an eerie quality to AI-generated prose. There's something off about it, even if you can't immediately tell what it is. If there are specific AI tells, and I'm using those tells right now, then how do you know I actually wrote this? Nabeel S. Qureshi, a former visiting scholar of AI at the Mercatus Center at George Mason University, was among the first to point out the suspected use of AI in Nazir's story. For Qureshi, the first two sentences were proof enough. They say the grove still hums at noon. Not the bees' neat industry or the clean rasp of cutlass on vibe, but a belly sound -- as if the earth swallows a shout and holds it there. "In general, AI writing has a particular rhythm that I've learned to pick up on which is hard to describe," Qureshi told me via email. "There's a spectrum from 'AI helped me edit' to 'AI wrote this' -- this case reads to me like the latter end of that, though of course I don't know for sure." The problem is that even when AI use is widely suspected, none of us really know for sure. In a statement, Commonwealth Foundation director-general Razmi Farook said the organization is aware of allegations regarding AI in the prizewinning stories, including Nazir's. Farook said all writers who submitted work for the prize are asked whether they're sending in original, unpublished work, and that all shortlisted writers have personally stated no AI was used to help them draft their stories. "Until a sufficient tool or process to reliably detect the use of AI emerges that can also grapple with the challenges pertaining to working with unpublished fiction, the Foundation and the Commonwealth Short Story Prize must operate on the principle of trust," Farook said. Granta, for its part, ran Nazir's story through Claude "and asked whether it was AI-generated," publisher Sigrid Rausing said in a statement. "The response was long, concluding that it was 'almost certainly not produced unaided by a human.'" But Claude isn't an AI detection tool, it's a chatbot powered by a large language model. Though AI tools are often better than human readers at detecting LLM-produced prose -- or at least those that judge literary prizes -- Granta's statement implied that they had gone to the source to ask whether the story in question had indeed been produced by AI, which demonstrates that perhaps the magazine itself does not understand how AI works either. "It may be that the judges have now awarded a prize to an instance of AI plagiarism -- we don't yet know, and perhaps we never will know," Rausing said. Publications are increasingly being tricked into running AI-generated stories, some of them "written" by "authors" who don't actually exist. There was even suspicion that Nazir himself was fake -- though author Kevin Jared Hosein, a previous Commonwealth Short Story Prize winner, confirmed that Nazir is a real person, and shared messages he recently exchanged with Nazir about the suspicions of AI use in his story. Nazir also published a poetry collection in 2018.) Nazir did not respond to The Verge's request for comment. In March, Hachette pulled the publication of Mia Ballard's horror novel Shy Girl after its author was accused of using AI, though Ballard denied using it and blamed a for-hire editor. There's also the question of whether there's any acceptable way for authors and journalists to use AI. LLM-generated prose is obviously verboten, but what about using AI for idea generation, or for research? What about AI transcription services? At what point does reliance on these tools mean the work is no longer your own? This week, Polish author Olga Tokarczuk admitted she uses AI to help with her creative process -- the other end of the AI-use spectrum Qureshi mentioned, but alarming to readers who admired a writer who had won the Nobel Prize in Literature. "I often simply throw into the machine an idea with the prompt: 'Darling, how could we beautifully elaborate this?'" said Tokarczuk, who was awarded literature's highest honor in 2018: "Even though I know about its hallucinations and numerous factual errors in the fields of quantitative economics or factual data, I have to admit that in the fluid field of literary fiction, this technology is an asset with unbelievable leverage. At the same time, I feel an acute human grief over an era that is disappearing never to return. I'm heartbroken by the departure of traditional literature written in isolation over months, a work conceived in the mind of a single conscious individual. In all of this, I'm damn mournful for Balzac, Cioran, and the inimitable Nabokov, because in spite of my enthusiasm, I don't believe that any modern chat has managed to speak in their exquisite manner." Tokarczuk's comments, which were delivered in Polish at a recent event in Poznań, had the misfortune of going viral around the same time of the Commonwealth Prize controversy. (We had her remarks translated into English by a human.) But she's far more ambivalent about AI than the headlines surrounding the event would suggest. Tokarczuk clarified her AI use in a three-point statement shared with Lit Hub in which she explained that she didn't use AI to write her forthcoming book but does use it for "faster documenting and checking of facts," though she independently verifies the information herself. "I am sometimes inspired by dreams," she continued, "but before this sentence too is cornered and torn to pieces by the experts, I hasten to report that they are my own dreams." The uproar over Tokarczuk's initial comments -- and the need she felt to explain herself -- speaks to a greater, not entirely unjustified paranoia in publishing over the use of AI. LLM-generated prose may be the new normal, but is that what anyone wants? Thousands of people threatened to boycott Barnes & Noble after CEO James Daunt said he had no problem selling AI-written books, so long as the books contained disclaimers specifying they hadn't been written by a person. Daunt later walked back his comments, but not entirely. "Book banning is a clear and present danger, so we are very careful with demands to ban any books," he told the Los Angeles Times, while also making sure "not to sell AI generated books that masquerade to be by real authors." None of this, however, explains the uncanny quality of AI-generated work, or what distinguishes bad LLM-produced prose from bad human writing. When I ran Nazir's story through Pangram, an AI- and plagiarism-detection software, it came back as 100 percent AI-generated. According to Pangram, the most obvious tells were Nazir's use of triads; the word "stubborn," which is six times as likely to appear in AI-generated text than that made by humans; and the phrase "as if it had," whose appearance is five times as likely. But here we have another list of three, written by me, a human. Dissatisfied, I ran an unpublished excerpt from my forthcoming book, which I am currently editing, through Pangram. One paragraph alone included two triads. (It is not a very good section of the book, which is why I'm editing it.) Pangram said the excerpt was 100 percent human-written, which is true, but I was still unsatisfied. I ran another excerpt -- a better one, I think -- and it said the same thing. When I ran the first chapter of Verge editor Kevin Nguyen's novel, Mỹ Documents, through Pangram, the result was the same. Pangram itself ran every Commonwealth Prize winner through its software, and found that two of the 2026 awardees, as well as the 2025 winner, appear to have been produced by AI. Human-produced work has some kind of ineffable quality, as does its inverse. Maybe AI-generated prose is like obscenity: You know it when you see it, even if you don't know why.