Author caught using AI in writing says he can't quit the technology despite fabricated quotes

AI put "synthetic quotes" in his book. But this author wants to keep using it.

Journalist and author Steven Rosenbaum has more reasons than most to distrust AI. His new book, The Future of Truth: How AI Reshapes Reality, is all about "how Truth is being bent, blurred, and synthesized" thanks to the "pressure of fast-moving, profit-driven AI." Yet a New York Times investigation this week found what Rosenbaum now acknowledges are "a handful of improperly attributed or synthetic quotes" linked to his use of AI tools while researching the book. These quotes include one that tech reporter Kara Swisher told the Times she "never said" and another that Northeastern University professor Lisa Feldman Barrett said "don't appear in [my] book, and they are also wrong." Rosenbaum is now working with editors on what he says is a full "citation audit" that will correct future editions. Speaking to Ars in the wake of the controversy, Rosenbaum says he "learned a lesson" and is "going to be much more suspicious" and "reticent to trust" AI outputs going forward. But he also can't tear himself away from the tools. Rather amazingly, Rosenbaum is not interested in going back to the AI-free research process he used to write previous books. "The idea of taking X years off [from AI] while it sorts itself out, and going back to, like, Microsoft Word ... it's just not in my nature," he told Ars. "[AI] is magical. Because it connects, it knits together ideas and gives you pathways to think about things that you're not going to come up with on your own." It's also magical in another way: Like J.R.R. Tolkien's One Ring, AI convinces many of those who use it that they can control its power properly. But can they? Slipping through the cracks Rosenbaum used AI tools during his writing process, he told me, "to surface ideas, locate articles, summarize themes, identify people or papers I might want to look into." He draws a hard line between this kind of research and the "actual reporting, narrative structure, interviews, arguments, and conclusions in the book," which he says are "entirely mine... There was never a time when AI was writing the book." In addition to chapters based on transcribed interviews that Rosenbaum says he conducted himself, The Future of Truth also includes more research-based chapters in which Rosenbaum said, "We're pulling facts and then knitting them together into a narrative." Tools like OpenAI's ChatGPT and Anthropic's Claude were used heavily to gather information, he said, with any nuggets mined by those tools tagged with a "this came from AI" warning in his notes. It's strangely creative and crafty and unusual in all these ways ... and then it betrays you in ways that are just really quite horrible. Steven Rosenbaum Those tagged AI-generated notes were then passed on to a fact-checker and two copy editors provided by the publisher, Rosenbaum said. Of the 285 outside citations in the book, six have been identified by the Times as problematic, including three so-called "synthetic quotes" that have no apparent source. (More examples could turn up as the book undergoes further review. And it's worth noting that most writers manage to include zero made-up quotes when they write a book.) "I think we did that [double-checking] incredibly effectively, but not a hundred percent," Rosenbaum told Ars. "We're doing the work, we're doing the best we can. We look at it, it looks right. We double-check it, and then we made a mistake." But the significant failure here highlights how the traditional fact-checking process might be ill-equipped to handle AI-assisted research. In the past, a fact-checker could be reasonably confident that any author quoting cited written works had simply copied down those quotes directly. These quotes would need to be checked, of course, but the fact that they're so easy to verify makes them less inherently suspicious. If AI tools are involved anywhere in the pipeline, though, that assumption goes out the window, and there needs to be an extra layer of skepticism that those quotes had been copied correctly or that they even exist at all. The widespread adoption of AI tools among writers of all stripes also comes at a time when financially pressured newsrooms and publishers are increasingly cutting copy editors and fact-checkers from their workflows. We've seen how AI-generated errors like this can make it into a published book even with a fact-checking layer. The risk of using those tools only increases for the many books that never go through any fact-checking before publication. Rosenbaum, for his part, agreed that "publishers are going to need new verification workflows designed specifically for AI-era research. That probably includes mandatory source tracing for quotations, better provenance tracking, clearer standards around AI-assisted research, and potentially (more irony here) AI tools that audit citations against primary materials." "I didn't set out to fabricate anything," Rosenbaum continued. "What happened is what increasingly happens to journalists, students, researchers, lawyers, and authors working with these systems every day: [There was] AI-generated information that looked authoritative, and some of it made its way too far downstream before being caught." Cursing at the machine Instances of prominent AI-generated errors are becoming distressingly common across a number of fields. Last year, the Chicago Sun-Times printed an advertorial summer reading list full of non-existent books dreamed up by AI. The New York Times recently had to issue a significant correction after published quotes attributed to Conservative leader Pierre Poilievre turned out to be "an A.I.-generated summary of his views." Publications including Wired and Business Insider pulled down multiple articles attributed to "Margaux Blanchard" that appeared to be AI-generated. Scholarly conferences have been beset by papers with hallucinated citations, and pre-print clearinghouse arXiv has recently implemented a zero-tolerance ban policy to try to stem the problem. Ars itself is not immune to this problem. Earlier this year, we retracted an article after a former reporter used an AI tool designed to extract verbatim quotes from a source's blog post -- but the tool instead generated fabricated versions of what the source actually wrote. (The latest version of our AI policy can be found here.) The irony of an author incorporating AI-generated falsehoods into a book on AI's reality-skewing effects is not lost on Rosenbaum. "I appreciate the book getting some attention, but this would not have been my choice about how to get it," he said. While that irony is "uncomfortable," he's quick to spin it as "also instructive. The fact that someone writing critically about AI and verification could still encounter these failures tells you how pervasive and persuasive these systems have become." Rosenbaum's own issue with AI "demonstrates the problem more vividly than any abstract argument could," he said. Perhaps. But if we accept this take, every avoidably obvious mess in the world might be a disguised good because it really helps illuminate the huge mistake. And that can't be right; sometimes "negligence" is just that. When asked directly how he could succumb to some of the AI-related problems his own book warns about, Rosenbaum described what sounds like a dysfunctional relationship with a charming charlatan. "As a writer, AI is often a delightful writing companion," Rosenbaum told me. "When I say 'writing companion,' I don't use that lightly. It's strangely creative and crafty and unusual in all these ways... and then it betrays you in ways that are just really quite horrible." Throughout our conversation, Rosenbaum frequently cited examples in which obvious AI errors left him enraged and literally cursing at the machine. Those date back to 2022, when Rosenbaum said he started experimenting with AI tools for "little research projects." At the time, he found AI answers "spectacularly useful" about 8 out of 10 times, with the remainder being confabulations that were "just not true." Despite these errors, he kept using the tools in his life and work. When we talked on Tuesday, Rosenbaum said he had recently asked an AI tool to extract his "no changes, verbatim" speaker's notes out of a slide deck so he could use them for an upcoming presentation. He was about to print those extracted notes when he realized that the LLM had actually rewritten his words despite his "very clear instructions for the robot." "And I say to it, 'Did you rewrite the words?' And it says, 'Well, I just made the language a little stronger.' Well, pardon me, but like, fuck you!" he said. Even in the face of these kinds of profanity-inducing errors, though, Rosenbaum still believes that AI tools are too efficient not to use. "The deck was 100 pages," Rosenbaum said. "To cut and paste page by page, the text from each page would have been an hour's worth of work, of mindless cutting and pasting. ChatGPT did it in about four seconds." To which the obvious retort might be: Yes, it was fast. But it was also wrong. Getting off the motorcycle The efficiency gains might be worth it when the only stakes are personal presentation notes. But The Future of Truth shows how the balance between AI's reliability and apparent speed should be weighed very differently when it comes to research that ends up in a published book. As we continued our conversation, I kept coming back to that accuracy/efficiency trade-off, which Rosenbaum seemed to recognize as a problem at some level. Even as he called AI's research help "magical" and "delightful," he described dealing with AI's confabulations and ignored directives as a "pernicious and exhausting" struggle. "It leaves you... uncomfortable almost any time you're using it," he said of its tendency to ignore clear instructions. "I've never fought with tech before this, honestly," he said at another point. "And I use it extensively." I've never been in a place where I thought the tech that I was using was both intoxicating and dangerous... Steven Rosenbaum Given the issues with his new book, I asked if the risk of introducing inaccuracies that you might not catch was really worth the perceived benefits. "I don't do drugs, and I don't drink, but I presume that that's kind of the question an addict asks when they're having one drink too many and they know they are," Rosenbaum said. "I've never been in a place where I thought the tech that I was using was both intoxicating and dangerous. And I wrote the book specifically to raise that concern, so if I end up being the poster child of not being aware of the guardrails, so be it." At one point, when discussing the relative risks and rewards of using AI, Rosenbaum noted that he rides a bicycle but wouldn't ride a motorcycle. "I know a motorcycle gets me places faster. I think it's dangerous and I might die. And that's why I don't own a motorcycle," he said. Rosenbaum made it clear that using AI was the relatively safe "bicycle" option in this analogy. I responded that the supercharged efficiency and catastrophic risk inherent in using AI made it feel a bit more like the motorcycle. Rosenbaum said "that might be fair" and thanked me for "sharpening" his analogy. I then asked the obvious question: Are you going to keep riding the motorcycle? "Can I get back to you on that?" he said.

We Asked the 'Future of Truth' Author to Explain How He Used AI. It Didn't Go Well

Earlier this month, WIRED published an excerpt from Steve Rosenbaum's buzzy new book, The Future of Truth, which looks at how artificial intelligence warps people's sense of reality. Shortly thereafter, The New York Times reported that the book contained over a half-dozen made-up or misattributed quotes. In a statement, Rosenbaum, who has a master's degree in "truth" from New York University, admitted that he had accidentally included "a handful" of "improperly attributed or synthetic" quotes. In an ironic twist, the veracity of a book about how AI impacts truth was now under intense scrutiny because of how its author had used AI. After the Times story broke, WIRED took another look at our 1,450-word excerpt. The fact-checking team had reviewed it prior to publication, and we reconfirmed that its quotes and facts were accurate. But WIRED's generative AI editorial policy prohibits the publication of AI-generated and AI-edited writing, and a reader email calling out the excerpt as being "blatantly AI-written" raised further questions about the extent to which Rosenbaum had used AI tools. In The Future of Truth's acknowledgement section, Rosenbaum writes that ChatGPT, Claude, NaturalReaders, ProWritingAid, and Grammarly had helped "refine and polish the presentation of [his] ideas." What, exactly, did that mean? WIRED ran the excerpt through several AI-detection services, including Pangram, GPTZero, and ZeroGPT. Each service suggested that it was either likely AI-generated, or AI-generated with high confidence. But AI-detection tools are fallible, and can return inaccurate readings. So WIRED's head of research emailed Rosenbaum directly to ask if and how he had used AI to write the excerpt. He wrote back: "Like many writers working today, I used AI tools during parts of the research and editorial development process for the book, including source discovery, brainstorming, structural feedback, and language refinement." But, he stressed, "the ideas, reporting, arguments, and final authorship are mine, and the WIRED excerpt was not generated by AI and then simply published as-is." He urged WIRED's editors to exercise caution trusting AI detection tools, noting that false positives can occur. At this point, WIRED's senior editors asked me to look into the episode, because I've covered AI slop in its various forms since 2024. My first step was to run the entire text of the book through Pangram's detection tool. (While all AI-detection tools have limitations, and can show false-positives, Pangram is the current gold standard.) It came back that the book appeared to be 53 percent AI-generated, with an additional 9 percent registering as likely AI-assisted. I called Rosenbaum and asked for a more detailed description of how he'd used AI to write the book, and whether he disputed Pangram's results. (BenBella Books, whose imprint published The Future of Truth, did not return requests for comment. Simon & Schuster, which distributes BenBella's books in the United States, declined to comment.) Rosenbaum would not weigh in on the accuracy of Pangram's results. In fact, he didn't want to talk about them at all. "I don't participate in that conversation," he said. "It's like saying, do you beat your wife? It's one of those accusations that there's no response to." He offered, instead, to broadly explain his editorial process. He says that at the beginning of the writing process, he used AI tools as search engines, helping him surface information for the more research-heavy sections of the book. To demonstrate how he might do this, he asked ChatGPT to describe me, then read the results out loud. The AI search more or less accurately described some of my prior stories, including work on AI-generated "zombie media sites." When I asked him to answer directly whether he had used AI to write or edit any of the passages in the book, he gave a winding answer. "No, that's not how you write. The answer is, you take material -- so, for example, the Kate response I just got, if I were to be writing about you, what would I do? Would I take it and paste it into Google Docs and then write around it and edit it up a little bit? Would I use the words 'zombie media sites' in quotes? Maybe. But I wouldn't say, oh great, that's answered my writing question for Kate, I'm going to stick that in my document and send it to my publisher." "But would you copy and paste it, and then edit it?" I asked. "Probably," he replied. "Did you do that in this book?" "I don't remember. You're looking for a smoking gun, and there isn't one." Rosenbaum said that writers who "wake up in the morning wanting to have ideas" are now "living in fear, and it's not healthy for democracy." Before I could ask what exactly that meant, he went on: "I talked to another author this morning who's literally got a book coming out going to the publisher in a month, and she's fucking terrified." I asked if that was because she used AI in the process of writing the book. "Of course. You say this with an accusatory tone, like she used a cheating tool." He mentioned a report that places AI adoption among journalists at 82 percent. He said that WIRED's generative AI policy is restrictive and hypothesized that our writers likely use AI in secret. He doubled down on his personal commitment to AI, noting that he still uses it every day. "If the only way for me to not end up with a mistake ever again is to literally stop using AI, that's just not realistic. If the answer is to stop writing, that's not out of the realm of possibility." I asked him whether he would rather stop writing than stop using AI in his writing process. "Yeah," he answered. Rosenbaum vacillated between acknowledging that AI use could cause problems ("I do not understand why it's my job as an author to play whack-a-mole with a multibillion-dollar company who puts hallucinations into their feed as a business practice") and repeatedly insisting that AI is indispensable, calling it the best writing partner he has ever had. He had simultaneously proved the thesis of his book -- AI was, clearly, causing rifts over what is authentic and what is not -- while undermining its credibility. When I told Rosenbaum that the way he used AI made me doubt the book's accuracy and overall quality, he again brought up the study about journalists' embrace of AI. "If 82 percent of journalists are using AI every day, then what you're saying is you now have anxiety about the accuracy and reliability of essentially everything that is in the current media ecosystem," he said. Well, yeah. I've been covering the rise of AI slop on the internet for years. It's pervasive. Skepticism is rational. But the study Rosenbaum cited, from the public relations software firm MuckRack, counts AI transcription and looking stuff up with ChatGPT as AI usage, which is substantively different from secretly incorporating AI-generated sentences into a draft. When I asked MuckRack for a more detailed breakdown of how it found that journalists used AI, the company noted that only a quarter of the writers it polled had used artificial-intelligence tools for writing assistance. The majority of writers are not there. At least not yet. Lines are constantly being drawn, scribbled out, and redrawn over appropriate AI use. I use an AI tool for transcription; I've also used Claude to create Freedom of Information Act requests. Some writers who oppose AI would say those use cases are wrong. Rosenbaum is a serial media entrepreneur. He was prescient on how important digital video and user-generated content would be on the internet. It's possible his way of operating could one day be the norm. Opinions around artificial intelligence in the media are already tilting toward increased acceptance. Earlier this year, WIRED spoke with tech reporters who openly embrace AI to write and edit stories. Independent journalist Alex Heath gives his reporting notes and other documents to an AI agent, which spins up his first draft. Some outlets are all in: Fortune, for example, is actively encouraging one of its reporters to cowrite stories with chatbots, and Business Insider permits its writers to use tools like ChatGPT on story drafts. A decade from now, newsrooms insisting that human writers do all the writing may look as quaint as getting a magazine delivered in print. For now, there's still some resistance. This year, the publishing house Hachette canceled plans to release a novel in the United States after Pangram indicated that it was largely AI-generated, even after the author denied that they had used AI in their writing. The New York Times severed ties with a freelancer for using AI. And recently, a firestorm of criticism ensued after a literary magazine published short stories that critics believe appeared to be AI-generated. WIRED, too, had already been deep in the process of revising its editorial guidelines around AI. One aspect that will remain the same is that published work cannot be written with AI. Given the uncertainty around Rosenberg's process, the excerpt was retracted as of Friday afternoon.

The literary world isn't prepared for AI

Since 2012, the British literary magazine Granta has published the regional winners of the annual Commonwealth Short Story Prize. This year, however, there was something off about one of the selections for the prestigious award: It appears to have been written by AI. Jamir Nazir's "The Serpent in the Grove" has many of the hallmarks of LLM-generated prose -- mixed metaphors, anaphora, lists of threes. (I'm aware this, too, is a list of threes, and I promise I wrote this post myself, unassisted, as I write all things.) I'll admit I was initially unconvinced by the allegation that Nazir's story had been generated by AI. I know people are using LLMs to help them write -- or to write for them, period -- but I've been wary of the sort of AI paranoia that has developed among my peers. Em dashes are supposedly an AI tell, as are the word "delve" and the aforementioned lists. Short, punchy sentences, too, especially when used to punctuate a succession of longer sentences. But I, a human being, have certainly used all of the above in my writing before. LLMs, after all, are trained on human writing. They mirror what they've been fed. And yet there's an eerie quality to AI-generated prose. There's something off about it, even if you can't immediately tell what it is. If there are specific AI tells, and I'm using those tells right now, then how do you know I actually wrote this? Nabeel S. Qureshi, a former visiting scholar of AI at the Mercatus Center at George Mason University, was among the first to point out the suspected use of AI in Nazir's story. For Qureshi, the first two sentences were proof enough. They say the grove still hums at noon. Not the bees' neat industry or the clean rasp of cutlass on vibe, but a belly sound -- as if the earth swallows a shout and holds it there. "In general, AI writing has a particular rhythm that I've learned to pick up on which is hard to describe," Qureshi told me via email. "There's a spectrum from 'AI helped me edit' to 'AI wrote this' -- this case reads to me like the latter end of that, though of course I don't know for sure." The problem is that even when AI use is widely suspected, none of us really know for sure. In a statement, Commonwealth Foundation director-general Razmi Farook said the organization is aware of allegations regarding AI in the prizewinning stories, including Nazir's. Farook said all writers who submitted work for the prize are asked whether they're sending in original, unpublished work, and that all shortlisted writers have personally stated no AI was used to help them draft their stories. "Until a sufficient tool or process to reliably detect the use of AI emerges that can also grapple with the challenges pertaining to working with unpublished fiction, the Foundation and the Commonwealth Short Story Prize must operate on the principle of trust," Farook said. Granta, for its part, ran Nazir's story through Claude "and asked whether it was AI-generated," publisher Sigrid Rausing said in a statement. "The response was long, concluding that it was 'almost certainly not produced unaided by a human.'" But Claude isn't an AI detection tool, it's a chatbot powered by a large language model. Though AI tools are often better than human readers at detecting LLM-produced prose -- or at least those that judge literary prizes -- Granta's statement implied that they had gone to the source to ask whether the story in question had indeed been produced by AI, which demonstrates that perhaps the magazine itself does not understand how AI works either. "It may be that the judges have now awarded a prize to an instance of AI plagiarism -- we don't yet know, and perhaps we never will know," Rausing said. Publications are increasingly being tricked into running AI-generated stories, some of them "written" by "authors" who don't actually exist. There was even suspicion that Nazir himself was fake -- though author Kevin Jared Hosein, a previous Commonwealth Short Story Prize winner, confirmed that Nazir is a real person, and shared messages he recently exchanged with Nazir about the suspicions of AI use in his story. Nazir also published a poetry collection in 2018.) Nazir did not respond to The Verge's request for comment. In March, Hachette pulled the publication of Mia Ballard's horror novel Shy Girl after its author was accused of using AI, though Ballard denied using it and blamed a for-hire editor. There's also the question of whether there's any acceptable way for authors and journalists to use AI. LLM-generated prose is obviously verboten, but what about using AI for idea generation, or for research? What about AI transcription services? At what point does reliance on these tools mean the work is no longer your own? This week, Polish author Olga Tokarczuk admitted she uses AI to help with her creative process -- the other end of the AI-use spectrum Qureshi mentioned, but alarming to readers who admired a writer who had won the Nobel Prize in Literature. "I often simply throw into the machine an idea with the prompt: 'Darling, how could we beautifully elaborate this?'" said Tokarczuk, who was awarded literature's highest honor in 2018: "Even though I know about its hallucinations and numerous factual errors in the fields of quantitative economics or factual data, I have to admit that in the fluid field of literary fiction, this technology is an asset with unbelievable leverage. At the same time, I feel an acute human grief over an era that is disappearing never to return. I'm heartbroken by the departure of traditional literature written in isolation over months, a work conceived in the mind of a single conscious individual. In all of this, I'm damn mournful for Balzac, Cioran, and the inimitable Nabokov, because in spite of my enthusiasm, I don't believe that any modern chat has managed to speak in their exquisite manner." Tokarczuk's comments, which were delivered in Polish at a recent event in Poznań, had the misfortune of going viral around the same time of the Commonwealth Prize controversy. (We had her remarks translated into English by a human.) But she's far more ambivalent about AI than the headlines surrounding the event would suggest. Tokarczuk clarified her AI use in a three-point statement shared with Lit Hub in which she explained that she didn't use AI to write her forthcoming book but does use it for "faster documenting and checking of facts," though she independently verifies the information herself. "I am sometimes inspired by dreams," she continued, "but before this sentence too is cornered and torn to pieces by the experts, I hasten to report that they are my own dreams." The uproar over Tokarczuk's initial comments -- and the need she felt to explain herself -- speaks to a greater, not entirely unjustified paranoia in publishing over the use of AI. LLM-generated prose may be the new normal, but is that what anyone wants? Thousands of people threatened to boycott Barnes & Noble after CEO James Daunt said he had no problem selling AI-written books, so long as the books contained disclaimers specifying they hadn't been written by a person. Daunt later walked back his comments, but not entirely. "Book banning is a clear and present danger, so we are very careful with demands to ban any books," he told the Los Angeles Times, while also making sure "not to sell AI generated books that masquerade to be by real authors." None of this, however, explains the uncanny quality of AI-generated work, or what distinguishes bad LLM-produced prose from bad human writing. When I ran Nazir's story through Pangram, an AI- and plagiarism-detection software, it came back as 100 percent AI-generated. According to Pangram, the most obvious tells were Nazir's use of triads; the word "stubborn," which is six times as likely to appear in AI-generated text than that made by humans; and the phrase "as if it had," whose appearance is five times as likely. But here we have another list of three, written by me, a human. Dissatisfied, I ran an unpublished excerpt from my forthcoming book, which I am currently editing, through Pangram. One paragraph alone included two triads. (It is not a very good section of the book, which is why I'm editing it.) Pangram said the excerpt was 100 percent human-written, which is true, but I was still unsatisfied. I ran another excerpt -- a better one, I think -- and it said the same thing. When I ran the first chapter of Verge editor Kevin Nguyen's novel, Mỹ Documents, through Pangram, the result was the same. Pangram itself ran every Commonwealth Prize winner through its software, and found that two of the 2026 awardees, as well as the 2025 winner, appear to have been produced by AI. Human-produced work has some kind of ineffable quality, as does its inverse. Maybe AI-generated prose is like obscenity: You know it when you see it, even if you don't know why.

The Second Reckoning Over AI Writing

Steven Rosenbaum has decided that the real villain behind the bogus quotes in his book is a chatbot. Earlier this week, The New York Times reported that The Future of Truth, Rosenbaum's much-discussed book about how AI shapes reality, contains more than half a dozen fake or misattributed quotes. Rosenbaum pinned some of them on his use of AI. He claimed responsibility for the errors and said he was investigating what went wrong. By the time I spoke with him on Thursday, though, he was pointing his finger elsewhere. ChatGPT "fucked up the book," Rosenbaum said. Rosenbaum, a media entrepreneur and the executive director of the Sustainable Media Center, said he came to rely on AI tools as both a resource and a conversation partner while he worked on the book (which he also notes in the book's acknowledgements). During our conversation, Rosenbaum struggled to reconcile AI's sometimes staggering capacities with its penchant for head-scratching hallucinations -- such as an imaginary quote from the tech journalist Kara Swisher that he included in the book without verifying it. In recent days, he has come to feel "seduced and betrayed" by AI, suggesting at one point that it might have undermined him on purpose. "Depending on your paranoia level, it's either quirky or evil or sneaky," he said. It's been a rough week for human authorship all around. On Monday, a viral post showed a Nobel-winning novelist seemingly admitting to using AI to sharpen her story ideas, before later claiming she had been misunderstood. On Tuesday, allegations mounted that the Trinidadian author Jamir Nazir had used AI to write "The Serpent in the Grove," which won the Commonwealth Short Story Prize. By Wednesday, two of the other five prize winners had come under similar scrutiny. (The Commonwealth Foundation, which administers the prize, initially said in a statement that it had confirmed that none of the winning writers used AI. On Friday, it issued another statement saying it "takes seriously the allegations" and was reviewing the evidence.) Read: This literary AI scandal changes everything Since ChatGPT arrived, automated writing has become ubiquitous: A recent working paper estimated that more than half of all new books released on Amazon now contain AI-generated text. Chatbots' prose has generally been good enough to fool schoolteachers and inflate Amazon product ratings -- not to earn glowing blurbs from prominent authors or win literary prizes. Recently, something has changed. As AI tools have improved and gone mainstream, the technology has penetrated intellectual spaces once thought to be fortified against its advances. This spate of scandals is forcing a fresh reckoning over what to do about the crisis. One response has been to call for a redoubling of efforts to root out AI writing and reinforce the stigma against it. If shame won't stop people from using AI to do the hard work of writing, maybe ridicule will. In Defector, Patrick Redford derided the "pathetic behavior" of writers who use AI. "You idiots!" he wrote. "Those models are the enemy!" Treating any use of AI in serious writing as taboo is understandable. Up until now, it's been relatively easy to use the hallmarks of AI-generated prose as a proxy for shoddy writing and thinking. Maybe we can keep that up a while longer. As I read The Future of Truth, I ran across an unusual amount of clunky repetition, formulaic transitions, and perplexing passages. One particularly tinny paragraph begins, "As we delve deeper into the mechanisms of misinformation, it's essential to understand how it not only proliferates but also profits." I ran the 146-word passage through Pangram, an AI-detection tool that is imperfect but reputed to be less flawed, at least, than some others. It registered the writing as 100 percent AI-generated. When I asked Rosenbaum whether he had let AI write any parts of his book, he said, "Absolutely not." When I mentioned the Pangram result, he said, "I'm not going to get into that game." The bigger challenge may be that "AI writing" is not just one thing. There's a wide spectrum between text that is untouched by machine intelligence and writing that is concocted entirely by a chatbot. At the maximalist end, most of us can agree that a writer wouldn't deserve a prize for typing, "Write a haunting, 3,000-word literary short story set in Trinidad" into Claude and then slapping his name on whatever it spits out. On the minimalist side, it's presumably fine for a writer to do some Googling in the process of researching a piece that is otherwise entirely her own. Then again, what they find may still be imbued with AI: Google search is answering more questions directly via chatbot, and the results are turning up more AI-written web pages. Good information comes from primary sources, not synthetic text. Generic chatbots have been joined by purpose-built AI research and writing tools that can carry out complex tasks. A growing number of professional writers, following the lead of software developers, openly profess to incorporating them into their workflows. The tech reporter Alex Heath, for instance, trained a version of Claude Cowork to write in his style and crank out first drafts of his stories, as Wired reported in March. My own use of AI is comparatively primitive but worth disclosing here: In line with The Atlantic's internal guidelines, I sometimes use chatbots like a slightly smarter thesaurus, to suggest the most apt word to plug into a given sentence, and I occasionally ask them to suggest expert sources on a specific topic. I also use an AI-powered tool to transcribe interviews, backstopped by my own notes. Read: The human skill that eludes AI Exactly where to draw the line on acceptable uses of AI is not as obvious as it might seem. In Rosenbaum's case, the scandal can't just be that he used AI while working on his book, because he acknowledged that up front. He got in trouble because he used AI badly, failing to check its work on a task at which it is famously unreliable. Or consider that The New York Times, which has endured a spate of AI writing scandals, maintains two different standards. Its freelancers can use AI tools for "high-level brainstorming" and almost nothing else. Newsroom employees are encouraged to experiment with what the paper's guidelines tout as "a powerful tool that, like many technological advances before it, may be used in service of our mission." The leading trade group for book authors, the Authors Guild, eschews edicts but warns of the ethical risks of various AI uses. Condoning AI for research but forbidding any use of its prose might be the most intuitive stance. It is certainly the most convenient: We have no reliable way to tell when AI was used to brainstorm ideas, research facts, or help a writer shape the framing of a story. But as the neuroscientist Tim Requarth pointed out in Slate, it is those hidden uses of AI in the writing process that give rise to our most valid concerns. The real threat the technology poses is not the overuse of the word "delve" in academic papers or the profusion of strained metaphors in literary fiction. It's that we lose something essential when we outsource to machines the hard work of discovering the truth and interpreting the world around us (or, in the case of fiction, the worlds within us). It's that the biases embedded in language models trained on dubious sources and controlled by tech companies will seep into the narratives that shape our understanding of reality. Are we sure that using AI to turn a phrase is worse than using it to decide what to write about in the first place? If nothing else, the pileup of scandals should force us to think more precisely about what it is we fear from AI writing. If the problem were simply that it's bad, then its steady improvement would be cause for relief rather than alarm. On the contrary, the problem seems to be that AI tools are getting too good, at least superficially, and that people are placing too much faith in them. Even though Rosenbaum cursed ChatGPT, he told me he couldn't imagine giving it up. That feeling might pose a greater threat to writing than anything he lays out in his book.

Man Humiliated by His AI Use Says He Just Can't Quit

Even after being caught -- and viciously criticized -- for using AI to write his book about AI, one writer says he's not giving up on the tech. While many AI writing scandals center on the artistic sin of letting a machine dictate your creative process, the offense committed by Steven Rosenbaum, the author of "The Future of Truth: How AI Reshapes Reality," was a journalistic one. A recent investigation by The New York Times found that his book contained more than a half dozen fabricated or misattributed quotes, after certain individuals "quoted" in the book came forward to confirm that they never said what Rosenbaum claimed they did. These turned out to be AI hallucinations, with Rosenbaum admitting to the paper that he used tools like ChatGPT and Claude while researching, writing, and editing the book. Now, after the storm of controversy those extremely ironic revelations sparked -- the book, after all, is explicitly about how AI affects our shared notion of the truth -- Rosenbaum says he had "learned a lesson" and will be "much more suspicious" of AI outputs going forward. Except you really have to wonder what lesson he's really taking away from all this, because he also said he was never going back to the old-fashioned, AI-free writing process. "The idea of taking X years off [from AI] while it sorts itself out, and going back to, like, Microsoft Word... it's just not in my nature," Rosenbaum told Ars Technica in an interview in the wake of the debacle. "[AI] is magical. Because it connects, it knits together ideas and gives you pathways to think about things that you're not going to come up with on your own." Throughout the interview, Rosenbaum described his AI helpers in tellingly anthropomorphic fashion, including calling AI a "delightful writing companion." "When I say 'writing companion,' I don't use that lightly," he told the outlet. "It's strangely creative and crafty and unusual in all these ways... and then it betrays you in ways that are just really quite horrible." It was hard not to wonder about tech's addictive potential as Rosenbaum rationalized his AI habit with bizarre analogies. He compared weighing the tech's benefits and risks to what a drug addict or alcoholic might ask themselves, while asserting he's "never been in a place where I thought the tech that I was using was both intoxicating and dangerous." Rosenbaum also compared using AI to his decision to ride a bicycle but not a motorcycle. When the interviewer questioned Rosenbaum's framing of AI as the safer "bicycle" option, since AI's accelerated productivity comes with a clear risk of errors that would make it seem more like a motorcycle, Rosenbaum had to concede that the interviewer's point "might be fair." Rosenbaum isn't alone in his error. Scandals have erupted around a number of books and stories this year after they were accused of being written with the help of AI, including a horror novel that ended up being pulled by its publisher. Newsrooms have also been caught up in AI controversy, as when both the NYT and Ars published articles that accidentally included quotes that turned out to be AI summaries or fabrications instead of what the attributed person had said verbatim. But Rosenbaum's blunder is particularly egregious, and not only because of the topic of his book. This is meant to be a work of non-fiction bolstered by commentary from industry experts, that went through multiple rounds of editing and fact-checking, and was picked up by a major publisher. Has the gravity of this dawned on Rosenbaum? He didn't sound particularly critical of himself. "I think we did that [double-checking] incredibly effectively, but not a hundred percent," Rosenbaum told Ars. "We're doing the work, we're doing the best we can. We look at it, it looks right. We double-check it, and then we made a mistake."

Nonfiction Book Publishers Aren't Remotely Ready for AI

Steven Rosenbaum started writing his book The Future of Truth: How AI Reshapes Reality in 2022, around when ChatGPT launched. Initially he didn't use it at all, "But as the writing moved forward into 2023, 2024, it got better and I got better at using it," he said. "To be clear, it never wrote a page of the book," he added. "But it became a research partner. I would ask it for quotes on certain things, and it would deliver them. They would occasionally be spectacular, often serviceable, and then, in very odd ways, just staggeringly wrong." "I kept thinking, I'll be really careful, and I'll double-check everything," he said. In May, the New York Times reported that Rosenbaum had included "more than a half-dozen misattributed or fake quotes" in the book seemingly generated by AI. Rosenbaum, a media entrepreneur, had previously acknowledged that he'd used AI tools during the research, writing, and editing process, but the Times investigation was nevertheless mortifying -- for both Rosenbaum and his publisher, Simon & Schuster. The book-publishing industry had already been wrestling with the prospect of a flood of AI-authored texts in the fiction market, and now the Rosenbaum scandal was showing the way AI could blow a hole in the nonfiction sector, too. Nonfiction publishing is uniquely vulnerable to AI because the industry has long neglected to do anything to ensure the books it publishes are factually accurate. "People outside of the industry don't understand that, contractually, publishers are not obligated to fact-check," said Paul Bogaards, the longtime marketing and publicity executive at Knopf who now has his own PR firm. Fact-checking is not a service publishers will pay for, though they sometimes encourage authors to seek it out on their own dime. But fact-checking is expensive: Hiring an outside checker can cost between $7,000 to $10,000 per book, or even more depending on its length, which might not be feasible for an author with a modest advance. Worse, it seems publishers have no idea what to do about this glaring vulnerability. "We don't have systems in place," said literary agent Alia Hanna Habib. "For every contract, there is a conversation, and it never really feels like anyone has the right answer," said one editor at a major publishing house. Editors, writers, and agents say the problem is likely already rampant. "I feel like everyone is passing off AI work as their own and most of the time don't say anything about it," said a senior nonfiction editor at a major publishing house. "I brought this up last year because I had been finding so many errors in the books that I was editing. I have just been told over and over again that the publisher can't take on the responsibility of fact-checking or hiring a freelance fact-checker because that shifts the responsibility onto us." Asserted Rosenbaum, "Anyone who is a working writer today who sits in front of a computer, either doing longform or on deadline or at magazines, whatever the cadence of your work is, you're using AI one way or another at least in part because it is not only seductive as hell but it's really incredibly valuable." A lot of writers would beg to differ with that statement. But it does point to the fact that there is no industrywide standard on what AI usage, if any, is acceptable even as everyone I spoke to seems to agree that guardrails are necessary. "A lot of authors are well intentioned in their use of AI and don't want to rely on AI to generate work that they would then present as their own," said Todd Shuster, co-founder of the literary agency Aevitas. "But they might rely on AI for some research or ideas around the structure of the book or outline. And the author then sort of forgets or denies or suppresses the extent to which they relied on the AI for such research. And before you know it it's not only that they've looked to AI for assistance, they've actually generated texts that they're including in their proposals or manuscripts." (New York does not allow writers or editors to use AI tools for drafting, outlining, or editing.) Contracts contain clauses in which the author warrants, among other things, that they are the sole author of the work and that the work is original, indemnifying the publisher. "But as of now, most publishers don't have contractual language that's specific to AI concerns," said Habib. "Therefore, when AI allegations are levied, you're in a kind of difficult spot. There's some contractual language around saying that something has to be an original work and against plagiarism, and I would say AI work would fail that test, but others might disagree." "I've always felt that, despite my ethical holdups with AI, there should be a more in-depth conversation on what things it absolutely shouldn't be allowed to be involved with," said one editor at a major publishing house. "Nobody seems willing to have that conversation." An editor at another publishing house said their company sends authors a guide about not using AI in certain ways, "But it's not legally binding. Once you're at the editing stage, this is basically a business of trust." Agents, too, are in an ambiguous position. "People are saying, 'Oh, what if we just have the writer declare that they haven't used AI?' That's a slippery slope, because what is the legal recourse for that?" said one literary agent. "And if they used it just for research, is that fine? Theoretically, that's how Rosenbaum used it, but that still led to hallucinations and inaccuracies." At the Gernert Company, literary agent Chris Parris-Lamb said they have "talked about putting something on the submissions page of our website that we do not represent work in which AI has been used to generate the text, in whole or in part. But we're not asking people to sign an affidavit when they submit their manuscript, and, unfortunately, shame is not the deterrent it should be." Some agents and editors are keeping this front of mind as they sign new clients, spending more time vetting writers for their human expertise. "If I'm getting you a six-figure book advance, I don't want you to be putting it into ChatGPT," said a top literary agent. "Go to the fucking library." Multiple people noted that a fact-checker would almost certainly have caught the fabricated and misattributed quotes in Rosenbaum's book, some only a Google search away. Rosenbaum said his book had three rounds of proofreading but that he did not hire a fact-checker, something that "was never suggested to me by the publisher," he said. (He has since personally hired two fact-checkers, who are working on a new version of the book.) He added that the Times article "sounds like the book is rife with errors and in fact if you count them, it's five, and of the five, one is a citation error. So it's four, which my guess in nonfiction is below the average. But I wrote a book about truth, and I got four things wrong, and the Times wrote about it, so I'm going to be the poster boy for now." Even full transparency about AI usage could pose a problem. "My belief is that unless there's full disclosure of this, it's improper. And even when there's full disclosure, there's the risk that you as an author, if you're including AI-generated content, are actually engaging in copyright infringement, because the LLMs often spew out significant amounts of prose that are verbatim copies of prior published work," said Shuster. "If those works aren't in the public domain, the defense of fair use is flimsy if not inapposite. These are the challenges facing us." AI-detection tools have become more effective over the years but are still not entirely reliable in their assessments, sometimes mistaking human writing for computer-generated work. "As a representative for authors, I really worry about the prospect of false positives," said Parris-Lamb. Still, "There's increased openness" to these tools, said Shuster, who has been consulting for Pangram, a start-up whose namesake AI-detection tool purports to be 99.98 percent accurate. Shuster has been introducing publishers and other companies to the software. "I don't know how feasible it is to just run every manuscript through one of these programs," said Parris-Lamb. "It's probably still going to be something that's done when suspicions are raised." And "anything that costs money, publishing is allergic to," one literary agent noted. "As usual, I think the onus is going to fall on the authors and agents."