AI Hallucinations Flood Libraries as Chatbots Invent Academic Papers That Don't Exist

Librarians can't keep up with bad AI

Generative artificial intelligence continues to have a problem with hallucinations. Although many responses to user queries are largely accurate, programs like ChatGPT, Google Gemini, and Microsoft Copilot are still prone to offering made-up information and facts. As bad as that is on its own, the issue is further complicated by a tendency for these AI programs to produce seemingly reputable, yet wholly imaginary, sources. But as annoying as that is for millions of users, it's becoming a major issue for the people trusted to provide reliable, real information: librarians. "For our staff, it is much harder to prove that a unique record doesn't exist," Sarah Falls, a research engagement librarian at the Library of Virginia, told Scientific American. Falls estimated that around 15 percent of all the reference questions received by her staff are written by generative AI, some of which include imaginary citations and sources. This increased burden placed on librarians and institutions is so bad that even organizations like the International Committee of the Red Cross are putting people on notice about the problem. "A specific risk is that generative AI tools always produce an answer, even when the historical sources are incomplete or silent," the ICRC cautioned in a public notice earlier this month. "Because their purpose is to generate content, they cannot indicate that no information exists; instead, they will invent details that appear plausible but have no basis in the archival record." Instead of asking a program like ChatGPT for a list of ICRC reports, the organization suggests you engage directly with their publicly available information catalogue and scholarly archives. The same strategy should be extended to any institution. Unfortunately, until more people understand the fallibility of generative AI, the burden will remain on human archivists. "We'll likely also be letting our users know that we must limit how much time we spend verifying information," Falls warned. There's a good reason why librarians remained an integral component in societies for thousands of years. Unlike generative AI, they're trained to think critically, search for answers, and most importantly, admit when they're wrong.

Librarians Aren't Hiding Secret Books From You That Only AI Knows About

Everyone knows that AI chatbots like ChatGPT, Grok, and Gemini can often hallucinate sources. But for the folks tasked with helping the public find books and journal articles, the fake AI bullshit is really taking its toll. Librarians sound absolutely exhausted by the requests for titles that don't exist, according to a new post from Scientific American. The magazine spoke with Sarah Falls, the chief of researcher engagement at the Library of Virginia, who estimates that about 15% of all emailed reference questions that they receive are generated by AI chatbots like ChatGPT. And the requests often include questions about fake citations. What's more, Falls suggests that people don't seem to believe librarians when they explain that a given record doesn't exist, a trend that's been reported elsewhere like 404 Media. Many people really believe their stupid chatbot over a human who specializes in finding reliable information day in and day out. A recent post from the International Committee of the Red Cross (ICRC) titled, "Important notice: AI generated archival reference," provides more evidence that librarians are just exhausted with it all. "If a reference cannot be found, this does not mean that the ICRC is withholding information. Various situations may explain this, including incomplete citations, documents preserved in other institutions, orâ€" increasinglyâ€"AI-generated hallucinations," the organization said. "In such cases, you may need to look into the administrative history of the reference to determine whether it corresponds to a genuine archival source." The year seems to have been filled with examples of fake books and journal articles created with AI. A freelance writer for the Chicago Sun-Times generated a summer reading list for the newspaper with 15 books to recommend. But ten of the books didn't exist. The first report from Health Secretary Robert F. Kennedy Jr.'s so-called Make America Healthy Again commission was released in May. A week later, reporters at NOTUS published their findings after going through all of the citations. At least seven didn't exist. You can't blame everything on AI. Papers have been retracted for giving fake citations since long before ChatGPT or any other chatbot came on the scene. Back in 2017, a professor at Middlesex University found at least 400 papers citing a non-existent research paper that was essentially the equivalent of filler text. The citation: Van der Geer, J., Hanraads, J.A.J., Lupton, R.A., 2010. The art of writing a scientific article. J Sci. Commun. 163 (2) 51-59. It's gibberish, of course. The citation seems to have been included in many lower quality papersâ€"likely due to laziness and sloppiness rather than an intent to deceive. But it's a safe bet that any authors of those pre-AI papers would have probably been embarrassed about their inclusion. The thing about AI tools is that too many humans have come to believe our chatbots are more trustworthy than humans. Why might users trust their AI over humans? For one thing, part of the magic trick that AI pulls is speaking in an authoritative voice. Who are you going to believe, the chatbot you're using all day or some random librarian on the phone? The other problem might have something to do with the fact that people develop what they believe are reliable tricks for making AI more reliable. Some people even think that adding things like "don't hallucinate" and "write clean code" to their prompt will make sure their AI only gives the highest quality output. If that actually worked, we imagine companies like Google and OpenAI would just add that to every prompt for you. If it does work, boy, have we got a lifehack for all the tech companies currently terrified of the AI bubble bursting.

Librarians Dumbfounded as People Keep Asking for Materials That Don't Exist

Librarians, and the books they cherish, are already fight a losing battle for our attention spans with all kinds of tech-enabled brainrot. Now, in a further assault to their sanity, AI models are generating so much slop that students and researchers keep coming into libraries and asking for journals, books, and records that don't exist, Scientific American reports. In a statement from the International Committee of the Red Cross spotted by the magazine, the humanitarian organization cautioned that AI chatbots like ChatGPT, Gemini, and Copilot are prone to generating fabricated archival references. "These systems do not conduct research, verify sources, or cross-check information," the ICRC, which maintains a vast library and archives, said in the warning. "They generate new content based on statistical patterns, and may therefore produce invented catalogue numbers, descriptions of documents, or even references to platforms that have never existed." Library of Virginia chief of researcher engagement Sarah Falls told SciAm that the AI inventions are wasting the time of librarians who are asked to hunt down nonexistent records. Fifteen percent of emailed reference questions that Fall's library receives, she claims, are now ChatGPT-generated, which include hallucinated primary source documents and published works. "For our staff, it is much harder to prove that a unique record doesn't exist," Falls added. Other librarians and researchers have spoken out about AI's effects on their profession. "This morning I spent time looking up citations for a student," wrote one user on Bluesky who identified themselves as a scholarly communications librarian. "By the time I got to the third (with zero results), I asked where they got the list, and the student admitted they were from Google's AI summary." "As a librarian who works with researchers," another wrote, "can confirm this is true." AI companies have put a heavy focus on creating powerful "reasoning" models aimed at researchers that can conduct a vast amount of research off a few prompts. OpenAI released its agentic model for conducting "deep research" in February, which it claims to do "at the level of a research analyst." At the time, OpenAI claimed it hallucinated at a lower rate than its other models, but admitted it struggled with separating "authoritative information from rumors," and conveying uncertainty when it presented the information. The ICRC warned about that pernicious flaw in its statement. AIs "cannot indicate that no information exists," it stated. "Instead, they will invent details that appear plausible but have no basis in the archival record." Though AI's hallucinatory habit is well known by now, and though no one in the AI industry has made particularly impressive progress in clamping down on it, the tech continues to run amok in academic research. Scientists and researchers, who you'd hope to be as empirical and skeptical as possible, are being caught left and right submitting papers filled with AI-fabricated citations. The field of AI research itself, ironically, is drowning in a flood of AI-written papers as some academics publish upwards of one hundred shoddily-written studies a year. Since nothing happens in a vacuum, the authentic, human-written sources and papers are now being drowned out. "Because of the amount of slop being produced, finding records that you KNOW exist but can't necessarily easily find without searching, has made finding real records that much harder," lamented a researcher on Bluesky.

AI Is Inventing Academic Papers That Don't Exist -- And They're Being Cited in Real Journals

A Complete History of the R-Word, the MAGA Movement's Favorite Slur As the fall semester came to a close, Andrew Heiss, an assistant professor in the Department of Public Management and Policy at the Andrew Young School of Policy Studies at Georgia State University, was grading coursework from his students when he noticed something alarming. As is typical for educators these days, Heiss was following up on citations in papers to make sure that they led to real sources -- and weren't fake references supplied by an AI chatbot. Naturally, he caught some of his pupils using generative artificial intelligence to cheat: not only can the bots help write the text, they can supply alleged supporting evidence if asked to back up claims, attributing findings to previously published articles. But, as with attorneys who have been caught generating briefs with AI because a model offered false legal precedents, students can end up with plausible-sounding footnotes pointing to academic articles and journals that don't exist. That in itself wasn't unusual, however. What Heiss came to realize in the course of vetting these papers was that AI-generated citations have now infested the world of professional scholarship, too. Each time he attempted to track down a bogus source in Google Scholar, he saw that dozens of other published articles had relied on findings from slight variations of the same made-up studies and journals. "There have been lots of AI-generated articles, and those typically get noticed and retracted quickly," Heiss tells Rolling Stone. He mentions a paper retracted earlier this month, which discussed the potential to improve autism diagnoses with an AI model and included a nonsensical infographic that was itself created with a text-to-image model. "But this hallucinated journal issue is slightly different," he says. That's because articles which include references to nonexistent research material -- the papers that don't get flagged and retracted for this use of AI, that is -- are themselves being cited in other papers, which effectively launders their erroneous citations. This leads to students and academics (and any large language models they may ask for help) identifying those "sources" as reliable without ever confirming their veracity. The more these false citations are unquestioningly repeated from one article to the next, the more the illusion of their authenticity is reinforced. Fake citations have turned into a nightmare for research librarians, who by some estimates are wasting up to 15 percent of their work hours responding to requests for nonexistent records that ChatGPT or Google Gemini alluded to. Heiss also noticed that the AI-generated notes could be convincing to a reader because they included the names of living academics and titles that closely resemble existing literature. In some cases, he found, the citation led him to an actual author, yet the heading of the article and the journal were both fabricated -- they just sounded similar to work the author has published in the past and a real periodical that covers such topics. "The AI-generated things get propagated into other real things, so students see them cited in real things and assume they're real, and get confused as to why they lose points for using fake sources when other real sources use them," he says. "Everything looks real and above-board." Since LLMs have become commonplace tools, academics have warned that they threaten to undermine our grasp on data by flooding the zone with fraudulent content. The psychologist and cognitive scientist Iris van Rooij has argued that the emergence of AI "slop" across scholarly resources portends nothing less than "the destruction of knowledge." In July, she and others in related fields signed an open letter calling on universities to resist the hype and marketing in order to "safeguard higher education, critical thinking, expertise, academic freedom, and scientific integrity." The authors claimed that schools have "coerced" faculty into using AI or allowing it in their classes, and they asked for a more rigorous, comprehensive analysis of whether it can have any useful role in education at all. Anthony Moser, a software engineer and technologist, was among those who foresaw how chatbots could eventually hollow out educational institutions. "I'm imagining an instructor somewhere making a syllabus with ChatGPT, assigning reading from books that don't exist," he wrote in a post on Bluesky in 2023, less than a year after the AI model first came out. "But the students don't notice, because they are asking ChatGPT to summarize the book or write the essay." This month, Moser reshared that post, commenting: "I wish it had taken longer for this to become literally true." Moser tells Rolling Stone that to even claim LLMs "hallucinate" fictional publications misunderstands the threat they pose to our comprehension of the world, because the term "implies that it's different from the normal, correct perception of reality." But the chatbots are "always hallucinating," he says. "It's not a malfunction. A predictive model predicts some text, and maybe it's accurate, maybe it isn't, but the process is the same either way. To put it another way: LLMs are structurally indifferent to truth." "LLMs are pernicious because they're essentially polluting the information ecosystem upstream," Moser adds. "Nonexistent citations show up in research that's sloppy or dishonest, and from there get into other papers and articles that cite them, and papers that cite those, and then it's in the water," he says, likening this content to like harmful, long-lasting chemicals: "hard to trace and hard to filter out, even when you're trying to avoid it." Moser calls the problem "the entirely foreseeable outcome of deliberate choices," with those who raised objections "ignored or overruled." But AI can't take all the blame. "Bad research isn't new," Moser points out. "LLMs have amplified the problem dramatically, but there was already tremendous pressure to publish and produce, and there were many bad papers using questionable or fake data, because higher education has been organized around the production of knowledge-shaped objects, measured in citations, conferences, and grants." Craig Callender, a philosophy professor at the University of California San Diego and president of the Philosophy of Science Association, agrees with that assessment, observing that "the appearance of legitimacy to non-existent journals is like the logical end product of existing trends." There are already journals, he explains, that accept spurious articles for profit, or biased ghost-written research meant to benefit the industry that produced it. "The 'swamp' in scientific publishing is growing," he says. "Many practices make existing journals [or] articles that aren't legitimate look legitimate. So the next step to non-existent journals is horrifying but not too surprising." Adding AI to the mix means that "swamp" is growing fast, Callender says. "For instance, all of this gets compounded in a nearly irreversible way with AI-assisted Google searches. These searches will only reinforce the appearance that these journals exist, just as they currently reinforce a lot of disinformation." All of which contributes to a feeling among researchers that they're being buried in an avalanche of slop, with limited capacity to sift through it. "It's been incredibly disheartening for faculty, I think fairly universally, especially as fake content gets accidentally enshrined in public research databases," says Heiss. "It's hard to work back up the citation chain to see where claims originated." Of course, many aren't even trying to do that -- which is why the phony stuff has been so widely disseminated. It's almost as if the uncritical and naive adoption of AI has made us more credulous and sapped our critical thinking at the precise moment we should be on guard against its evolving harms. In fact, someone may be toiling away on a (real) study of that phenomenon right now.