AI Hallucinations in Legal Filings: Morgan & Morgan Warns Lawyers of Consequences

AI making up cases can get lawyers fired, scandalized law firm warns

Morgan & Morgan -- which bills itself as "America's largest injury law firm" that fights "for the people" -- learned the hard way this month that even one lawyer blindly citing AI-hallucinated case law can risk sullying the reputation of an entire nationwide firm. In a letter shared in a court filing, Morgan & Morgan's chief transformation officer, Yath Ithayakumar, warned the firms' more than 1,000 attorneys that citing fake AI-generated cases in court filings could be cause for disciplinary action, including "termination." "This is a serious issue," Ithayakumar wrote. "The integrity of your legal work and reputation depend on it." Morgan & Morgan's AI troubles were sparked in a lawsuit claiming that Walmart was involved in designing a supposedly defective hoverboard toy that allegedly caused a family's house fire. Despite being an experienced litigator, Rudwin Ayala, the firm's lead attorney on the case, cited eight cases in a court filing that Walmart's lawyers could not find anywhere except on ChatGPT. These "cited cases seemingly do not exist anywhere other than in the world of Artificial Intelligence," Walmart's lawyers said, urging the court to consider sanctions. So far, the court has not ruled on possible sanctions. But Ayala was immediately dropped from the case and was replaced by his direct supervisor, T. Michael Morgan, Esq. Expressing "great embarrassment" over Ayala's fake citations that wasted the court's time, Morgan struck a deal with Walmart's attorneys to pay all fees and expenses associated with replying to the errant court filing, which Morgan told the court should serve as a "cautionary tale" for both his firm and "all firms." Reuters found that lawyers improperly citing AI-hallucinated cases have scrambled litigation in at least seven cases in the past two years. Some lawyers have been sanctioned, including an early case last June fining lawyers $5,000 for citing chatbot "gibberish" in filings. And in at least one case in Texas, Reuters reported, a lawyer was fined $2,000 and required to attend a course on responsible use of generative AI in legal applications. But in another high-profile incident, Michael Cohen, Donald Trump's former lawyer, avoided sanctions after Cohen accidentally gave his own attorney three fake case citations to help his defense in his criminal tax and campaign finance litigation. In a court filing, Morgan explained that Ayala was solely responsible for the AI citations in the Walmart case. No one else involved " had any knowledge or even notice" that the errant court filing "contained any AI-generated content, let alone hallucinated content," Morgan said, insisting that had he known, he would have required Ayala to independently verify all citations. "The risk that a Court could rely upon and incorporate invented cases into our body of common law is a nauseatingly frightening thought," Morgan said, "deeply" apologizing to the court while acknowledging that AI can be "dangerous when used carelessly." Further, Morgan said, it's clear that his firm must work harder to train attorneys on AI tools the firm has been using since November 2024 that were intended to support -- not replace -- lawyers as they researched cases. Despite the firm supposedly warning lawyers that AI can hallucinate or fabricate information, Ayala shockingly claimed that he "mistakenly" believed that the firm's "internal AI support" was "fully capable" of not just researching but also drafting briefs. "This deeply regrettable filing serves as a hard lesson for me and our firm as we enter a world in which artificial intelligence becomes more intertwined with everyday practice," Morgan told the court. "While artificial intelligence is a powerful tool, it is a tool which must be used carefully. There are no shortcuts in law." Andrew Perlman, dean of Suffolk University's law school, advocates for responsible AI use in court and told Reuters that lawyers citing ChatGPT or other AI tools without verifying outputs is "incompetence, just pure and simple." Morgan & Morgan declined Ars' request to comment. Law firm makes changes to prevent AI citations Morgan & Morgan wants to make sure that no one else at the firm makes the same mistakes that Ayala did. In the letter sent to all attorneys, Ithayakumar reiterated that AI cannot be solely used to dependably research cases or draft briefs, as "AI can generate plausible responses that may be entirely fabricated information." "As all lawyers know (or should know), it has been documented that AI sometimes invents case law, complete with fabricated citations, holdings, and even direct quotes," his letter said. "As we previously instructed you, if you use AI to identify cases for citation, every case must be independently verified." While Harry Surden, a law professor who studies AI legal issues, told Reuters that "lawyers have always made mistakes," he also suggested that an increasing reliance on AI tools in the legal field requires lawyers to increase AI literacy to fully understand "the strengths and weaknesses of the tools." (A July 2024 Reuters survey found that 63 percent of lawyers have used AI and 12 percent use it regularly, after experts signaled an AI-fueled paradigm shift in the legal field in 2023.) At Morgan & Morgan, it has become clear in 2025 that better AI training is needed across its nationwide firm. Morgan told the court that the firm's technology team and risk management members have met to "discuss and implement further policies to prevent another occurrence in the future." Additionally, a checkbox acknowledging AI's potential for hallucinations was added, and it must be clicked before any attorney at the firm can access the internal AI platform. "Further, safeguards and training are being discussed to protect against any errant uses of artificial intelligence," Morgan told the court. Whether these efforts will help Morgan & Morgan avoid sanctions is unclear, but Ithayakumar suggested that on par with sanctions might be the reputational loss to the firm's or any individual lawyer's credibility. "Blind reliance on AI is equivalent to citing an unverified case," Ithayakumar told lawyers, saying that it is their "responsibility and ethical obligation" to verify AI outputs. "Failure to comply with AI verification requirements may result in court sanctions, professional discipline, discipline by the firm (up to and including termination), and reputational harm. Every lawyer must stay informed of the specific AI-related rules and orders in the jurisdictions where they practice and strictly adhere to these obligations."

AI 'Hallucinations' in Court Papers Spell Trouble for Lawyers

(Reuters) - U.S. personal injury law firm Morgan & Morgan sent an urgent email this month to its more than 1,000 lawyers: Artificial intelligence can invent fake case law, and using made-up information in a court filing could get you fired. A federal judge in Wyoming had just threatened to sanction two lawyers at the firm who included fictitious case citations in a lawsuit against Walmart. One of the lawyers admitted in court filings last week that he used an AI program that "hallucinated" the cases and apologized for what he called an inadvertent mistake. AI's penchant for generating legal fiction in case filings has led courts around the country to question or discipline lawyers in at least seven cases over the last two years, and created a new high-tech headache for litigants and judges, Reuters found. The Walmart case stands out because it involves a well-known law firm and a big corporate defendant. But examples like it have cropped up in all kinds of lawsuits since chatbots like ChatGPT ushered in the AI era, highlighting a new litigation risk. A Morgan & Morgan spokesperson did not respond to a request for comment. Walmart declined to comment. The judge has not yet ruled whether to discipline the lawyers in the Walmart case, which involved an allegedly defective hoverboard toy. Advances in generative AI are helping reduce the time lawyers need to research and draft legal briefs, leading many law firms to contract with AI vendors or build their own AI tools. Sixty-three percent of lawyers surveyed by Reuters' parent company Thomson Reuters last year said they have used AI for work, and 12% said they use it regularly. Generative AI, however, is known to confidently make up facts, and lawyers who use it must take caution, legal experts said. AI sometimes produces false information, known as "hallucinations" in the industry, because the models generate responses based on statistical patterns learned from large datasets rather than by verifying facts in those datasets. Attorney ethics rules require lawyers to vet and stand by their court filings or risk being disciplined. The American Bar Association told its 400,000 members last year that those obligations extend to "even an unintentional misstatement" produced through AI. The consequences have not changed just because legal research tools have evolved, said Andrew Perlman, dean of Suffolk University's law school and an advocate of using AI to enhance legal work. "When lawyers are caught using ChatGPT or any generative AI tool to create citations without checking them, that's incompetence, just pure and simple," Perlman said. 'LACK OF AI LITERACY' In one of the earliest court rebukes over attorneys' use of AI, a federal judge in Manhattan in June 2023 fined two New York lawyers $5,000 for citing cases that were invented by AI in a personal injury case against an airline. A different New York federal judge last year considered imposing sanctions in a case involving Michael Cohen, the former lawyer and fixer for Donald Trump, who said he mistakenly gave his own attorney fake case citations that the attorney submitted in Cohen's criminal tax and campaign finance case. Cohen, who used Google's AI chatbot Bard, and his lawyer were not sanctioned, but the judge called the episode "embarrassing." In November, a Texas federal judge ordered a lawyer who cited nonexistent cases and quotations in a wrongful termination lawsuit to pay a $2,000 penalty and attend a course about generative AI in the legal field. A federal judge in Minnesota last month said a misinformation expert had destroyed his credibility with the court after he admitted to unintentionally citing fake, AI-generated citations in a case involving a "deepfake" parody of Vice President Kamala Harris. Harry Surden, a law professor at the University of Colorado's law school who studies AI and the law, said he recommends lawyers spend time learning "the strengths and weaknesses of the tools." He said the mounting examples show a "lack of AI literacy" in the profession, but the technology itself is not the problem. "Lawyers have always made mistakes in their filings before AI," he said. "This is not new." (Reporting by Sara Merken in New York; Editing by David Bario and Aurora Ellis)

AI 'hallucinations' in court papers spell trouble for lawyers

(Reuters) - U.S. personal injury law firm Morgan & Morgan sent an urgent email this month to its more than 1,000 lawyers: Artificial intelligence can invent fake case law, and using made-up information in a court filing could get you fired. A federal judge in Wyoming had just threatened to sanction two lawyers at the firm who included fictitious case citations in a lawsuit against Walmart. One of the lawyers admitted in court filings last week that he used an AI program that "hallucinated" the cases and apologized for what he called an inadvertent mistake. AI's penchant for generating legal fiction in case filings has led courts around the country to question or discipline lawyers in at least seven cases over the last two years, and created a new high-tech headache for litigants and judges, Reuters found. The Walmart case stands out because it involves a well-known law firm and a big corporate defendant. But examples like it have cropped up in all kinds of lawsuits since chatbots like ChatGPT ushered in the AI era, highlighting a new litigation risk. A Morgan & Morgan spokesperson did not respond to a request for comment. Walmart declined to comment. The judge has not yet ruled whether to discipline the lawyers in the Walmart case, which involved an allegedly defective hoverboard toy. Advances in generative AI are helping reduce the time lawyers need to research and draft legal briefs, leading many law firms to contract with AI vendors or build their own AI tools. Sixty-three percent of lawyers surveyed by Reuters' parent company Thomson Reuters last year said they have used AI for work, and 12% said they use it regularly. Generative AI, however, is known to confidently make up facts, and lawyers who use it must take caution, legal experts said. AI sometimes produces false information, known as "hallucinations" in the industry, because the models generate responses based on statistical patterns learned from large datasets rather than by verifying facts in those datasets. Attorney ethics rules require lawyers to vet and stand by their court filings or risk being disciplined. The American Bar Association told its 400,000 members last year that those obligations extend to "even an unintentional misstatement" produced through AI. The consequences have not changed just because legal research tools have evolved, said Andrew Perlman, dean of Suffolk University's law school and an advocate of using AI to enhance legal work. "When lawyers are caught using ChatGPT or any generative AI tool to create citations without checking them, that's incompetence, just pure and simple," Perlman said. 'LACK OF AI LITERACY' In one of the earliest court rebukes over attorneys' use of AI, a federal judge in Manhattan in June 2023 fined two New York lawyers $5,000 for citing cases that were invented by AI in a personal injury case against an airline. A different New York federal judge last year considered imposing sanctions in a case involving Michael Cohen, the former lawyer and fixer for Donald Trump, who said he mistakenly gave his own attorney fake case citations that the attorney submitted in Cohen's criminal tax and campaign finance case. Cohen, who used Google's AI chatbot Bard, and his lawyer were not sanctioned, but the judge called the episode "embarrassing." In November, a Texas federal judge ordered a lawyer who cited nonexistent cases and quotations in a wrongful termination lawsuit to pay a $2,000 penalty and attend a course about generative AI in the legal field. A federal judge in Minnesota last month said a misinformation expert had destroyed his credibility with the court after he admitted to unintentionally citing fake, AI-generated citations in a case involving a "deepfake" parody of Vice President Kamala Harris. Harry Surden, a law professor at the University of Colorado's law school who studies AI and the law, said he recommends lawyers spend time learning "the strengths and weaknesses of the tools." He said the mounting examples show a "lack of AI literacy" in the profession, but the technology itself is not the problem. "Lawyers have always made mistakes in their filings before AI," he said. "This is not new." (Reporting by Sara Merken in New York; Editing by David Bario and Aurora Ellis)

AI 'hallucinations' in court papers spell trouble for lawyers

Advances in generative AI are helping reduce the time lawyers need to research and draft legal briefs, leading many law firms to contract with AI vendors or build their own AI tools. Sixty-three percent of lawyers surveyed by Reuters' parent company Thomson Reuters last year said they have used AI for work, and 12% said they use it regularly.US personal injury law firm Morgan & Morgan sent an urgent email this month to its more than 1,000 lawyers: Artificial intelligence can invent fake case law, and using made-up information in a court filing could get you fired. A federal judge in Wyoming had just threatened to sanction two lawyers at the firm who included fictitious case citations in a lawsuit against Walmart. One of the lawyers admitted in court filings last week that he used an AI program that "hallucinated" the cases and apologised for what he called an inadvertent mistake. AI's penchant for generating legal fiction in case filings has led courts around the country to question or discipline lawyers in at least seven cases over the last two years, and created a new high-tech headache for litigants and judges, Reuters found. The Walmart case stands out because it involves a well-known law firm and a big corporate defendant. But examples like it have cropped up in all kinds of lawsuits since chatbots like ChatGPT ushered in the AI era, highlighting a new litigation risk. A Morgan & Morgan spokesperson did not respond to a request for comment. Walmart declined to comment. The judge has not yet ruled whether to discipline the lawyers in the Walmart case, which involved an allegedly defective hoverboard toy. Advances in generative AI are helping reduce the time lawyers need to research and draft legal briefs, leading many law firms to contract with AI vendors or build their own AI tools. Sixty-three percent of lawyers surveyed by Reuters' parent company Thomson Reuters last year said they have used AI for work, and 12% said they use it regularly. Generative AI, however, is known to confidently make up facts, and lawyers who use it must take caution, legal experts said. AI sometimes produces false information, known as "hallucinations" in the industry, because the models generate responses based on statistical patterns learned from large datasets rather than by verifying facts in those datasets. Attorney ethics rules require lawyers to vet and stand by their court filings or risk being disciplined. The American Bar Association told its 400,000 members last year that those obligations extend to "even an unintentional misstatement" produced through AI. The consequences have not changed just because legal research tools have evolved, said Andrew Perlman, dean of Suffolk University's law school and an advocate of using AI to enhance legal work. "When lawyers are caught using ChatGPT or any generative AI tool to create citations without checking them, that's incompetence, just pure and simple," Perlman said. 'Lack of AI literacy' In one of the earliest court rebukes over attorneys' use of AI, a federal judge in Manhattan in June 2023 fined two New York lawyers $5,000 for citing cases that were invented by AI in a personal injury case against an airline. A different New York federal judge last year considered imposing sanctions in a case involving Michael Cohen, the former lawyer and fixer for Donald Trump, who said he mistakenly gave his own attorney fake case citations that the attorney submitted in Cohen's criminal tax and campaign finance case. Cohen, who used Google's AI chatbot Bard, and his lawyer were not sanctioned, but the judge called the episode "embarrassing." In November, a Texas federal judge ordered a lawyer who cited nonexistent cases and quotations in a wrongful termination lawsuit to pay a $2,000 penalty and attend a course about generative AI in the legal field. A federal judge in Minnesota last month said a misinformation expert had destroyed his credibility with the court after he admitted to unintentionally citing fake, AI-generated citations in a case involving a "deepfake" parody of Vice President Kamala Harris. Harry Surden, a law professor at the University of Colorado's law school who studies AI and the law, said he recommends lawyers spend time learning "the strengths and weaknesses of the tools." He said the mounting examples show a "lack of AI literacy" in the profession, but the technology itself is not the problem. "Lawyers have always made mistakes in their filings before AI," he said. "This is not new."

Large Law Firm Sends Panicked Email as It Realizes Its Attorneys Have Been Using AI to Prepare Court Documents

The law firm Morgan & Morgan has rushed out a stern email to its attorneys after two of them were caught citing fake court cases invented by an AI model, Reuters reports. Sent earlier this month to all of its over 1,000 lawyers, the email warns at length about the tech's proclivity for hallucinating. But the pros of the tech, apparently, still outweigh the cons; rather than banning AI usage -- something that plenty of organizations have done -- Morgan & Morgan leadership take the middle road and give the usual spiel about please double-checking your work to ensure it's not totally made-up nonsense. "As we previously instructed you, if you use AI to identify cases for citation, every case must be independently verified," the email reads. "The integrity of your legal work and reputation depend on it." Last week, a federal judge in Wyoming admonished two Morgan & Morgan lawyers for citing at least nine instance of fake case law in court filings submitted in January. Threatened with sanctions, the embarrassed lawyers blamed an "internal AI tool" for the mishap, and pleaded the judge for mercy. "When lawyers are caught using ChatGPT or any generative AI tool to create citations without checking them, that's incompetence, just pure and simple," Andrew Perlman, dean of Suffolk University's law school and an advocate of using AI in legal work, told Reuters. The judge hasn't decided whether he'll punish the lawyers yet, per Reuters. Nonetheless, it's an enormous embarrassment for the relatively well-known firm, especially given the occasion. The lawsuit in question is against the world's largest company, Walmart, alleging that a hoverboard that the retailer sold was responsible for a fire that burned down the plaintiff's home. Now, the corporate lawyers are probably cackling to themselves in a backroom somewhere, their opponents having shot themselves in the foot so spectacularly. Anyone familiar with the shortcomings inherent to large language models could've seen something like this happening from a mile away. And according to Reuters, the tech's dubious usage in legal settings has already led to lawyers being questioned or disciplined in at least seven cases in the past two years. It's not just the hallucinations that are so pernicious -- it's how authoritatively the AI models lie to you. That, and the fact that anything that promises to automate a task more often than not tends to induce the person using it to let their guard down, a problem that's become pretty apparent in self-driving cars, for example, or in news agencies that have experimented with using AI to summarize stories or to assist with reporting. And so organizations can tell their employees to double-check their work all they want, but the fact remains that screw-ups like these will keep happening. To address the issue, Morgan & Morgan are requiring attorneys to acknowledge they're aware about the risks associated with AI usage by clicking a little box added to its AI tool, The Register reported. We're sure that'll do the trick.

Judge fines lawyers in Walmart lawsuit over fake, AI-generated cases

Feb 25 (Reuters) - Three lawyers suing Walmart in a personal injury lawsuit must pay fines totaling $5,000 for citing fake cases generated by artificial intelligence in a court filing, a federal judge ruled. The lawyers, including two from national law firm Morgan & Morgan and one from a smaller firm, had an ethical obligation to ensure that the cases they cited were real, U.S. District Judge Kelly Rankin in Wyoming said in his sanctions order, opens new tab on Monday. "As attorneys transition to the world of AI, the duty to check their sources and make a reasonable inquiry into existing law remains unchanged," Rankin wrote. Judges across the country have questioned or disciplined a growing number of lawyers over the past two years for including AI-generated cases and quotations in court documents. Examples have cropped up in at least nine lawsuits since chatbots like ChatGPT ushered in the AI era, highlighting a new litigation risk. Rankin imposed a $3,000 fine on Rudwin Ayala of Morgan & Morgan, who earlier this month apologized and said he used an internal AI program that "hallucinated" the cases, which he incorporated into a filing in the case. The judge also removed Ayala from the lawsuit, and he imposed fines of $1,000 each on T. Michael Morgan and local counsel Taly Goody of Goody Law Group for not doing enough to ensure the accuracy of the filing that Ayala drafted. Ayala, Morgan and a representative for Morgan & Morgan did not immediately respond to requests for comment on Tuesday. Morgan is the son of Morgan & Morgan founder John Morgan, according to past media reports. Goody said in an email that she did not receive the filing to review before it was submitted, and pointed to an earlier explanation she provided to the court. Walmart declined to comment. It has denied the allegations in the underlying lawsuit, which accuses the retailer of selling a defective hoverboard toy that allegedly caught fire and burned down the plaintiffs' house. The case is scheduled to go to trial next month. In deciding sanctions, the judge said he weighed the fact that lawyers "have been on notice" about generative AI's ability to hallucinate cases. He said Ayala's "honesty and candor" and steps taken by Morgan & Morgan to head off future incidents helped lead to a less severe punishment. "When done right, AI can be incredibly beneficial for attorneys and the public," Rankin wrote. "The instant case is simply the latest reminder to not blindly rely on AI platforms' citations regardless of profession." In a separate case, a federal magistrate judge in Indiana last week recommended a $15,000 fine for a lawyer who submitted three briefs that had non-existent case citations generated by AI and referred him to the chief judge for possible additional discipline. Our Standards: The Thomson Reuters Trust Principles., opens new tab Suggested Topics:Government Sara Merken Thomson Reuters Sara Merken reports on the business of law, including legal innovation and law firms in New York and nationally.

Lawyer faces $15,000 fine for using AI-generated fake cases in court filing

Facepalm: Another instance of an attorney using generative AI to file briefs containing non-existent cases has led to a judge recommending a $15,000 fine for his actions. That's more than three times what two lawyers and their law firm were fined in 2023 for doing the same thing. When representing HooserVac LLC in a lawsuit over its retirement fund in October 2024, Indiana attorney Rafael Ramirez included case citations in three separate briefs. The court could not locate these cases as they had been fabricated by ChatGPT. In December, US Magistrate Judge for the Southern District of Indiana Mark J. Dinsmore ordered Ramirez to appear in court and show cause as to why he shouldn't be sanctioned for the errors. "Transposing numbers in a citation, getting the date wrong, or misspelling a party's name is an error," the judge wrote. "Citing to a case that simply does not exist is something else altogether. Mr Ramirez offers no hint of an explanation for how a case citation made up out of whole cloth ended up in his brief. The most obvious explanation is that Mr Ramirez used an AI-generative tool to aid in drafting his brief and failed to check the citations therein before filing it." Ramirez admitted that he used generative AI, but insisted he did not realize the cases weren't real as he was unaware that AI could generate fictitious cases and citations. He also confessed to not complying with Federal Rule of Civil Procedure 11. This states that claims being made must be based on evidence that currently exists, or there is a strong likelihood that evidence will be found to support them through further investigation or discovery. The rule is supposed to encourage attorneys to perform due diligence before filing cases. "I swear these are real cases, your honor. ChatGPT assured me they were" Ramirez says he has since taken legal education courses on the use of AI in law, and continues to use AI tools. But the judge said his "failure to comply with that most basic of requirements" makes his conduct "particularly sanctionable." Dinsmore added (via Bloomberg Law) that as Ramirez failed to provide competent representation and made several false statements to the court, he was being referred to the chief judge for any further disciplinary action. Dinsmore has recommended that Ramirez be sanctioned $5,000 for each of the three cases he filed containing the fabricated cases. This isn't the first case of a lawyer's reliance on AI proving misplaced. In June 2023, two lawyers and their law firm were fined $5,000 by a district judge in Manhattan for citing fake legal research generated by ChatGPT. In January, lawyers in Wyoming submitted nine cases to support an argument in a lawsuit against Walmart and Jetson Electric Bikes over a fire allegedly caused by a hoverboard. Eight of the cases had been hallucinated by ChatGPT.

Judge recommends $15K sanctions for AI slop court filing

A federal magistrate judge has recommended $15,000 in sanctions be imposed on an attorney who cited non-existent court cases concocted by an AI chatbot. In a report [PDF] filed last week, Mark J. Dinsmore, US Magistrate Judge for the Southern District of Indiana, recommends that attorney Rafael Ramirez, of Rio, Hondo, Texas, be "sanctioned $15,000 for his violations in this case - $5,000 for each of the three briefs filed by Mr Ramirez where he failed to appropriately verify the validity and accuracy of the case law he cited to the court and opposing counsel." Back on October 29, 2024, Ramirez cited three non-existent cases in a brief. Dinsmore in December ordered [PDF] Ramirez to appear in court and show cause why he should not be sanctioned. ... did not know that AI was capable of generating fictitious cases and citations "Transposing numbers in a citation, getting the date wrong, or misspelling a party's name is an error," the judge wrote. "Citing to a case that simply does not exist is something else altogether. Mr Ramirez offers no hint of an explanation for how a case citation made up out of whole cloth ended up in his brief. The most obvious explanation is that Mr Ramirez used an AI-generative tool to aid in drafting his brief and failed to check the citations therein before filing it." That indeed was the explanation offered during a January 3, 2025 hearing on the matter. "Mr Ramirez admitted that he had relied on programs utilizing generative artificial intelligence ('AI') to draft the briefs," Dinsmore's report states. "Mr Ramirez explained that he had used AI before to assist with legal matters, such as drafting agreements, and did not know that AI was capable of generating fictitious cases and citations." While Ramirez insisted he did not act in bad faith, he did acknowledge not fully complying with Federal Rule of Civil Procedure 11, which requires attorneys presenting material to the court to certify it is accurate to the best of their knowledge. Ramirez did not immediately respond to a request for comment. Dinsmore noted that while the penalty he has recommended is "at the higher end of the sanctions that have previously been imposed for similar conduct, Mr Ramirez's professed ignorance of the propensity of the AI tools he was using to 'hallucinate' citations is evidence that those lesser sanctions have been insufficient to deter the conduct." Indeed, ample recent reporting about artificial intelligence's tendency to make things up has not deterred lawyers from using chatbots to help draft documents. We reported a similar incident in Wyoming earlier this month, and another in New York. And this are just the court proceedings; who knows how models are being used in marketing and accountancy now, for instance, let alone legal work and programming. AI-generated falsehoods have even tripped up Minnesota Attorney General Keith Ellison in the case of Kohls et al v. Ellison et al. Defending a Minnesota ban on the dissemination of "deepfakes" designed to injure a political candidate or influence an election from First Amendment free speech challenge, Ellison submitted two expert reports from professors offering "background about artificial intelligence ('AI'), deepfakes, and the dangers of deepfakes to free speech and democracy." This proved problematic for Ellison because a declaration [PDF] from Jeff Hancock, professor of communication at Stanford University and director of the Stanford Social Media Lab, included two references to academic articles that did not exist. This did not please the court. ... fallen victim to the siren call of relying too heavily on AI "After reviewing plaintiffs' motion to exclude, Attorney General Ellison's office contacted Professor Hancock, who subsequently admitted [PDF] that his declaration inadvertently included citations to two non-existent academic articles, and incorrectly cited the authors of a third article," wrote US District Judge Laura M. Provinzino in a January 10, 2025 order [PDF] granting a motion to exclude Hancock's report. "These errors apparently originated from Professor Hancock's use of GPT-4o - a generative AI tool - in drafting his declaration." Hancock did not immediately respond to a request for comment. Judge Provinzino notes that it is ironic that "a credentialed expert on the dangers of AI and misinformation, has fallen victim to the siren call of relying too heavily on AI - in a case that revolves around the dangers of AI, no less." ®

AI hallucinations could get lawyers fired, law firm says

U.S. personal injury law firm Morgan & Morgan sent an urgent email this month to its more than 1,000 lawyers: Artificial intelligence can invent fake case law, and using made-up information in a court filing could get you fired. A federal judge in Wyoming had just threatened to sanction two lawyers at the firm who included fictitious case citations in a lawsuit against Walmart. One of the lawyers admitted in court filings last week that he used an AI program that "hallucinated" the cases and apologized for what he called an inadvertent mistake. AI's penchant for generating legal fiction in case filings has led courts around the country to question or discipline lawyers in at least seven cases over the past two years, and created a new high-tech headache for litigants and judges, Reuters found. The Walmart case stands out because it involves a well-known law firm and a big corporate defendant. But examples like it have cropped up in all kinds of lawsuits since chatbots like ChatGPT ushered in the AI era, highlighting a new litigation risk.