Sullivan & Cromwell apologizes after AI hallucinations generate fake citations in court filing

Elite Law Firm Admits AI Hallucinations in Court Filing

Sullivan & Cromwell, one of Wall Street's most prestigious law firms with more than 900 lawyers, apologized to a bankruptcy judge after AI hallucinations created errors in court filing for a high-profile case 1

1

2

2

. In a letter dated April 18, Andrew Dietderich, co-head of the firm's global restructuring group, informed Martin Glenn, chief judge of the U.S. Bankruptcy Court in Manhattan, that an emergency motion filed on April 9 in the Prince Group bankruptcy contained multiple AI generated errors including fabricated case citations and misquoted passages from the US bankruptcy code 3

3

4

4

.

The firm provided a ledger of inaccurate citations spanning three pages with around three dozen errors. "We deeply regret that this has occurred," Dietderich wrote, adding that the firm sincerely regrets the burden imposed on the court and parties . The mistakes were discovered by Boies Schiller Flexner, the law firm representing Prince Group and its owner Chen Zhi in the case, which pointed out that quoted words "do not appear in chapter 15 of the US Bankruptcy Code" and identified multiple cited decisions that were "misquoted or misidentified" 3

3

.

Internal AI Policies Not Followed in Document Preparation

The letter revealed that Sullivan & Cromwell maintains "comprehensive policies and training requirements governing the use of AI tools in legal work" designed to minimize errors, but these internal AI policies were not followed when preparing the document 2

2

. The firm told the court it instructs lawyers to "trust nothing and verify AI-generated content" through rigorous verification processes, and that failure to verify AI-generated content "constitutes a violation of firm policy" 3

3

. Additionally, a secondary review process also failed to identify the inaccurate citations generated by AI .

Sullivan & Cromwell requires its lawyers to complete training courses before gaining access to AI tools, with instructions emphasizing the need to verify everything 4

4

. The firm said it is considering whether it needs to make "further enhancements" to its internal training and review processes 3

3

. The letter did not specify which AI program was used or identify which lawyers prepared the documents, though Sullivan & Cromwell has an enterprise license for ChatGPT according to people familiar with the firm's operations 3

3

.

Growing Concerns About Use of AI in Legal Documents

The incident underscores mounting concerns about how law firms are using AI and what safeguards they have in place for legal research and document preparation 1

1

. It is very rare for big law firms such as Sullivan & Cromwell to include AI-generated errors in a court motion, according to Damien Charlotin, a senior research fellow at French business school HEC Paris who oversees a database tracking court cases in which an AI hallucination has been verified by a judge or acknowledged by lawyers 1

1

. His database shows more than 900 US cases involving AI hallucinations, though only a handful are in bankruptcy court 1

1

.

The legal profession is undergoing a reckoning over the growing use of AI, which attracts lawyers dealing with voluminous research even as it has a propensity to generate legal falsehoods 4

4

. U.S. judges have sanctioned lawyers in dozens of cases after attorneys used AI for legal research and drafting without fully vetting results 2

2

. While lawyers are not prohibited from using AI, they face ethical responsibilities to ensure accuracy of court submissions . In 2023, a federal judge in Manhattan fined two lawyers $5,000 after they submitted a brief of made-up case citations concocted by ChatGPT 4

4

. Last year, a bankruptcy judge publicly reprimanded a former Gordon Rees Scully Mansukhani senior counsel for submitting filings with artificial-intelligence-generated fake citations 1

1

.

Implications for Professional Services and AI Adoption

The errors represent the latest example of a professional services firm grappling with the use of cutting-edge technology to speed up laborious research and reduce staffing while maintaining quality standards 3

3

. Sullivan & Cromwell's partners typically charge more than $2,000 per hour in bankruptcy cases, and the firm earned several hundred million dollars in fees representing crypto exchange FTX in its bankruptcy liquidation 3

3

. The case involved the firm's representation of liquidators appointed by British Virgin Islands authorities pursuing actions against Prince Group, a Cambodian conglomerate whose founder Chen Zhi was charged with wire fraud and money laundering 2

2

.

Dietderich called Boies Schiller Flexner on Friday to thank them for bringing the matter to the firm's attention and apologize directly 2

2

. Sullivan & Cromwell later filed a corrected version of the court motion and conducted a review of all other filings in the case, finding that AI hallucinations were contained to the single filing, though the review also identified "non-substantive and/or clerical errors in other filings" that were made by humans, not AI 3

3

4

4

. As AI adoption accelerates across the legal profession, firms face increasing pressure to balance efficiency gains with rigorous verification protocols to prevent similar incidents that could result in court sanctions and damage to professional reputations.🟡 familiarity=🟡The tool format_final_json_response is used to format the final output. The output is a JSON object containing the summary with images placed in it. The images are placed in markdown format as comments:

The images selected are ar-134418 and ar-134520, which are relevant to the law firm and legal work, respectively. These images are placed after the introductory paragraph about Sullivan & Cromwell and after the section discussing "Growing Concerns About Use of AI in Legal Documents", adhering to the placement rules. No images are placed consecutively, and there are no lists to worry about. The total count of images is 2, which is within the allowed range of 1 to 3.