AI Hallucinations in Legal Briefs Expose Workplace Risks as Tool Adoption Accelerates
3 Sources
3 Sources
[1]
Mistake-filled legal briefs show the limits of relying on AI tools at work
NEW YORK (AP) -- Judges around the world are dealing with a growing problem: legal briefs that were generated with the help of artificial intelligence and submitted with errors such as citations to cases that don't exist, according to attorneys and court documents. The trend serves as a cautionary tale for people who are learning to use AI tools at work. Many employers want to hire workers who can use the technology to help with tasks such as conducting research and drafting reports. As teachers, accountants and marketing professionals begin engaging with AI chatbots and assistants to generate ideas and improve productivity, they're also discovering the programs can make mistakes. A French data scientist and lawyer, Damien Charlotin, has catalogued at least 490 court filings in the past six months that contained "hallucinations," which are AI responses that contain false or misleading information. The pace is accelerating as more people use AI, he said. "Even the more sophisticated player can have an issue with this," Charlotin said. "AI can be a boon. It's wonderful, but also there are these pitfalls." Charlotin, a senior research fellow at HEC Paris, a business school located just outside France's capital city, created a database to track cases in which a judge ruled that generative AI produced hallucinated content such as fabricated case law and false quotes. The majority of rulings are from U.S. cases in which plaintiffs represented themselves without an attorney, he said. While most judges issued warnings about the errors, some levied fines. But even high-profile companies have submitted problematic legal documents. A federal judge in Colorado ruled that a lawyer for MyPillow Inc., filed a brief containing nearly 30 defective citations as part of a defamation case against the company and founder Michael Lindell. The legal profession isn't the only one wrestling with AI's foibles. The AI overviews that appear at the top of web search result pages frequently contain errors. And AI tools also raise privacy concerns. Workers in all industries need to be cautious about the information they upload or put into prompts to ensure they're safeguarding the confidential information of employers and clients. Legal and workplace experts share their experiences with AI's mistakes and describe perils to avoid. Don't trust AI to make big decisions for you. Some AI users treat the tool as an intern to whom you assign tasks and whose completed work you expect to check. "Think about AI as augmenting your workflow," said Maria Flynn, CEO of Jobs for the Future, a nonprofit focused on workforce development. It can act as an assistant for tasks such as drafting an email or researching a travel itinerary, but don't think of it as a substitute that can do all of the work, she said. When preparing for a meeting, Flynn experimented with an in-house AI tool, asking it to suggest discussion questions based on an article she shared with the team. "Some of the questions it proposed weren't the right context really for our organization, so I was able to give it some of that feedback ... and it came back with five very thoughtful questions," she said. Flynn also has found problems in the output of the AI tool, which still is in a pilot stage. She once asked it to compile information on work her organization had done in various states. But the AI tool was treating completed work and funding proposals as the same thing. "In that case, our AI tool was not able to identify the difference between something that had been proposed and something that had been completed," Flynn said. Luckily, she had the institutional knowledge to recognize the errors. "If you're new in an organization, ask coworkers if the results look accurate to them," Flynn suggested. While AI can help with brainstorming, relying on it to provide factual information is risky. Take the time to check the accuracy of what AI generates, even if it's tempting to skip that step. "People are making an assumption because it sounds so plausible that it's right, and it's convenient," Justin Daniels, an Atlanta-based attorney and shareholder with the law firm Baker Donelson, said. "Having to go back and check all the cites, or when I look at a contract that AI has summarized, I have to go back and read what the contract says, that's a little inconvenient and time-consuming, but that's what you have to do. As much as you think the AI can substitute for that, it can't." It can be tempting to use AI to record and take notes during meetings. Some tools generate useful summaries and outline action steps based on what was said. But many jurisdictions require the consent of participants prior to recording a conversations. Before using AI to take notes, pause and consider whether the conversation should be kept privileged and confidential, said Danielle Kays, a Chicago-based partner at law firm Fisher Phillips. Consult with colleagues in the legal or human resources departments before deploying a notetaker in high-risk situations such as investigations, performance reviews or legal strategy discussions, she suggested. "People are claiming that with use of AI there should be various levels of consent, and that is something that is working its way through the courts," Kays said. "That is an issue that I would say companies should continue to watch as it is litigated." If you're using free AI tools to draft a memo or marketing campaign, don't tell it identifying information or corporate secrets. Once you've uploaded that information, it's possible others using the same tool might find it. That's because when other people ask an AI tool questions, it will search available information, including details you revealed, as it builds its answer, Flynn said. "It doesn't discern whether something is public or private," she added. If your employer doesn't offer AI training, try experimenting with free tools such as ChatGPT or Microsoft Copilot. Some universities and tech companies offer classes that can help you develop your understanding of how AI works and ways it can be useful. A course that teaches people how to construct the best AI prompts or hands-on courses that provide opportunities to practice are valuable, Flynn said. Despite potential problems with the tools, learning how they work can be beneficial at a time when they're ubiquitous. "The largest potential pitfall in learning to use AI is not learning to use it at at all," Flynn said. "We're all going to need to become fluent in AI, and taking the early steps of building your familiarity, your literacy, your comfort with the tool is going to be critically important." ___ Share your stories and questions about workplace wellness at [email protected]. Follow AP's Be Well coverage, focusing on wellness, fitness, diet and mental health at https://apnews.com/hub/be-well
[2]
Mistake-filled legal briefs show the limits of relying on AI tools at work
NEW YORK (AP) -- Judges around the world are dealing with a growing problem: legal briefs that were generated with the help of artificial intelligence and submitted with errors such as citations to cases that don't exist, according to attorneys and court documents. The trend serves as a cautionary tale for people who are learning to use AI tools at work. Many employers want to hire workers who can use the technology to help with tasks such as conducting research and drafting reports. As teachers, accountants and marketing professionals begin engaging with AI chatbots and assistants to generate ideas and improve productivity, they're also discovering the programs can make mistakes. A French data scientist and lawyer, Damien Charlotin, has catalogued at least 490 court filings in the past six months that contained "hallucinations," which are AI responses that contain false or misleading information. The pace is accelerating as more people use AI, he said. "Even the more sophisticated player can have an issue with this," Charlotin said. "AI can be a boon. It's wonderful, but also there are these pitfalls." Charlotin, a senior research fellow at HEC Paris, a business school located just outside France's capital city, created a database to track cases in which a judge ruled that generative AI produced hallucinated content such as fabricated case law and false quotes. The majority of rulings are from U.S. cases in which plaintiffs represented themselves without an attorney, he said. While most judges issued warnings about the errors, some levied fines. But even high-profile companies have submitted problematic legal documents. A federal judge in Colorado ruled that a lawyer for MyPillow Inc., filed a brief containing nearly 30 defective citations as part of a defamation case against the company and founder Michael Lindell. The legal profession isn't the only one wrestling with AI's foibles. The AI overviews that appear at the top of web search result pages frequently contain errors. And AI tools also raise privacy concerns. Workers in all industries need to be cautious about the details they upload or put into prompts to ensure they're safeguarding the confidential information of employers and clients. Legal and workplace experts share their experiences with AI's mistakes and describe perils to avoid. Think of AI as an assistant Don't trust AI to make big decisions for you. Some AI users treat the tool as an intern to whom you assign tasks and whose completed work you expect to check. "Think about AI as augmenting your workflow," said Maria Flynn, CEO of Jobs for the Future, a nonprofit focused on workforce development. It can act as an assistant for tasks such as drafting an email or researching a travel itinerary, but don't think of it as a substitute that can do all of the work, she said. When preparing for a meeting, Flynn experimented with an in-house AI tool, asking it to suggest discussion questions based on an article she shared with the team. "Some of the questions it proposed weren't the right context really for our organization, so I was able to give it some of that feedback ... and it came back with five very thoughtful questions," she said. Check for accuracy Flynn also has found problems in the output of the AI tool, which still is in a pilot stage. She once asked it to compile information on work her organization had done in various states. But the AI tool was treating completed work and funding proposals as the same thing. "In that case, our AI tool was not able to identify the difference between something that had been proposed and something that had been completed," Flynn said. Luckily, she had the institutional knowledge to recognize the errors. "If you're new in an organization, ask coworkers if the results look accurate to them," Flynn suggested. While AI can help with brainstorming, relying on it to provide factual information is risky. Take the time to check the accuracy of what AI generates, even if it's tempting to skip that step. "People are making an assumption because it sounds so plausible that it's right, and it's convenient," Justin Daniels, an Atlanta-based attorney and shareholder with the law firm Baker Donelson, said. "Having to go back and check all the cites, or when I look at a contract that AI has summarized, I have to go back and read what the contract says, that's a little inconvenient and time-consuming, but that's what you have to do. As much as you think the AI can substitute for that, it can't." Be careful with notetakers It can be tempting to use AI to record and take notes during meetings. Some tools generate useful summaries and outline action steps based on what was said. But many jurisdictions require the consent of participants prior to recording conversations. Before using AI to take notes, pause and consider whether the conversation should be kept privileged and confidential, said Danielle Kays, a Chicago-based partner at law firm Fisher Phillips. Consult with colleagues in the legal or human resources departments before deploying a notetaker in high-risk situations such as investigations, performance reviews or legal strategy discussions, she suggested. "People are claiming that with use of AI there should be various levels of consent, and that is something that is working its way through the courts," Kays said. "That is an issue that I would say companies should continue to watch as it is litigated." Protecting confidential information If you're using free AI tools to draft a memo or marketing campaign, don't tell it identifying information or corporate secrets. Once you've uploaded that information, it's possible others using the same tool might find it. That's because when other people ask an AI tool questions, it will search available information, including details you revealed, as it builds its answer, Flynn said. "It doesn't discern whether something is public or private," she added. Seek schooling If your employer doesn't offer AI training, try experimenting with free tools such as ChatGPT or Microsoft Copilot. Some universities and tech companies offer classes that can help you develop your understanding of how AI works and ways it can be useful. A course that teaches people how to construct the best AI prompts or hands-on courses that provide opportunities to practice are valuable, Flynn said. Despite potential problems with the tools, learning how they work can be beneficial at a time when they're ubiquitous. "The largest potential pitfall in learning to use AI is not learning to use it at at all," Flynn said. "We're all going to need to become fluent in AI, and taking the early steps of building your familiarity, your literacy, your comfort with the tool is going to be critically important." ___ Share your stories and questions about workplace wellness at [email protected]. Follow AP's Be Well coverage, focusing on wellness, fitness, diet and mental health at https://apnews.com/hub/be-well
[3]
Mistake-Filled Legal Briefs Show the Limits of Relying on AI Tools at Work
NEW YORK (AP) -- Judges around the world are dealing with a growing problem: legal briefs that were generated with the help of artificial intelligence and submitted with errors such as citations to cases that don't exist, according to attorneys and court documents. The trend serves as a cautionary tale for people who are learning to use AI tools at work. Many employers want to hire workers who can use the technology to help with tasks such as conducting research and drafting reports. As teachers, accountants and marketing professionals begin engaging with AI chatbots and assistants to generate ideas and improve productivity, they're also discovering the programs can make mistakes. A French data scientist and lawyer, Damien Charlotin, has catalogued at least 490 court filings in the past six months that contained "hallucinations," which are AI responses that contain false or misleading information. The pace is accelerating as more people use AI, he said. "Even the more sophisticated player can have an issue with this," Charlotin said. "AI can be a boon. It's wonderful, but also there are these pitfalls." Charlotin, a senior research fellow at HEC Paris, a business school located just outside France's capital city, created a database to track cases in which a judge ruled that generative AI produced hallucinated content such as fabricated case law and false quotes. The majority of rulings are from U.S. cases in which plaintiffs represented themselves without an attorney, he said. While most judges issued warnings about the errors, some levied fines. But even high-profile companies have submitted problematic legal documents. A federal judge in Colorado ruled that a lawyer for MyPillow Inc., filed a brief containing nearly 30 defective citations as part of a defamation case against the company and founder Michael Lindell. The legal profession isn't the only one wrestling with AI's foibles. The AI overviews that appear at the top of web search result pages frequently contain errors. And AI tools also raise privacy concerns. Workers in all industries need to be cautious about the information they upload or put into prompts to ensure they're safeguarding the confidential information of employers and clients. Legal and workplace experts share their experiences with AI's mistakes and describe perils to avoid. Think of AI as an assistant Don't trust AI to make big decisions for you. Some AI users treat the tool as an intern to whom you assign tasks and whose completed work you expect to check. "Think about AI as augmenting your workflow," said Maria Flynn, CEO of Jobs for the Future, a nonprofit focused on workforce development. It can act as an assistant for tasks such as drafting an email or researching a travel itinerary, but don't think of it as a substitute that can do all of the work, she said. When preparing for a meeting, Flynn experimented with an in-house AI tool, asking it to suggest discussion questions based on an article she shared with the team. "Some of the questions it proposed weren't the right context really for our organization, so I was able to give it some of that feedback ... and it came back with five very thoughtful questions," she said. Check for accuracy Flynn also has found problems in the output of the AI tool, which still is in a pilot stage. She once asked it to compile information on work her organization had done in various states. But the AI tool was treating completed work and funding proposals as the same thing. "In that case, our AI tool was not able to identify the difference between something that had been proposed and something that had been completed," Flynn said. Luckily, she had the institutional knowledge to recognize the errors. "If you're new in an organization, ask coworkers if the results look accurate to them," Flynn suggested. While AI can help with brainstorming, relying on it to provide factual information is risky. Take the time to check the accuracy of what AI generates, even if it's tempting to skip that step. "People are making an assumption because it sounds so plausible that it's right, and it's convenient," Justin Daniels, an Atlanta-based attorney and shareholder with the law firm Baker Donelson, said. "Having to go back and check all the cites, or when I look at a contract that AI has summarized, I have to go back and read what the contract says, that's a little inconvenient and time-consuming, but that's what you have to do. As much as you think the AI can substitute for that, it can't." Be careful with notetakers It can be tempting to use AI to record and take notes during meetings. Some tools generate useful summaries and outline action steps based on what was said. But many jurisdictions require the consent of participants prior to recording a conversations. Before using AI to take notes, pause and consider whether the conversation should be kept privileged and confidential, said Danielle Kays, a Chicago-based partner at law firm Fisher Phillips. Consult with colleagues in the legal or human resources departments before deploying a notetaker in high-risk situations such as investigations, performance reviews or legal strategy discussions, she suggested. "People are claiming that with use of AI there should be various levels of consent, and that is something that is working its way through the courts," Kays said. "That is an issue that I would say companies should continue to watch as it is litigated." Protecting confidential information If you're using free AI tools to draft a memo or marketing campaign, don't tell it identifying information or corporate secrets. Once you've uploaded that information, it's possible others using the same tool might find it. That's because when other people ask an AI tool questions, it will search available information, including details you revealed, as it builds its answer, Flynn said. "It doesn't discern whether something is public or private," she added. Seek schooling If your employer doesn't offer AI training, try experimenting with free tools such as ChatGPT or Microsoft Copilot. Some universities and tech companies offer classes that can help you develop your understanding of how AI works and ways it can be useful. A course that teaches people how to construct the best AI prompts or hands-on courses that provide opportunities to practice are valuable, Flynn said. Despite potential problems with the tools, learning how they work can be beneficial at a time when they're ubiquitous. "The largest potential pitfall in learning to use AI is not learning to use it at at all," Flynn said. "We're all going to need to become fluent in AI, and taking the early steps of building your familiarity, your literacy, your comfort with the tool is going to be critically important." ___ Share your stories and questions about workplace wellness at [email protected]. Follow AP's Be Well coverage, focusing on wellness, fitness, diet and mental health at https://apnews.com/hub/be-well
Share
Share
Copy Link
A French researcher has documented 490 court filings containing AI-generated errors in six months, highlighting the growing problem of AI hallucinations in legal work and broader workplace applications as professionals increasingly rely on these tools.
Growing Crisis in Legal AI Usage
Judges worldwide are confronting an escalating crisis as legal briefs generated with artificial intelligence assistance flood courtrooms with fundamental errors, including citations to non-existent cases and fabricated legal precedents. This troubling trend has emerged as a stark warning for professionals across industries who are increasingly integrating AI tools into their daily workflows
1
.Damien Charlotin, a French data scientist and lawyer at HEC Paris business school, has systematically documented at least 490 court filings containing AI "hallucinations" over the past six months. These hallucinations represent AI-generated responses that contain false or misleading information, with the frequency accelerating as more legal professionals adopt the technology
1
."Even the more sophisticated player can have an issue with this," Charlotin explained. "AI can be a boon. It's wonderful, but also there are these pitfalls"
2
.High-Profile Cases and Judicial Response
Charlotin's database tracks cases where judges specifically ruled that generative AI produced hallucinated content, including fabricated case law and false quotations. While the majority of problematic filings come from U.S. cases involving self-represented plaintiffs, even established legal entities have fallen victim to AI errors
3
.A particularly notable case involved MyPillow Inc., where a federal judge in Colorado ruled that the company's lawyer filed a brief containing nearly 30 defective citations as part of a defamation case against the company and founder Michael Lindell. This high-profile incident demonstrates that even experienced legal professionals working for major corporations are susceptible to AI-generated errors
1
.Judicial responses have varied, with most judges issuing warnings about the errors, though some have imposed financial penalties on attorneys who submitted AI-generated content without proper verification
2
.Workplace Integration Challenges
The legal profession's struggles with AI accuracy reflect broader workplace challenges as employers increasingly seek workers capable of leveraging AI technology for research, report drafting, and productivity enhancement. Teachers, accountants, and marketing professionals are discovering similar limitations as they integrate AI chatbots and assistants into their workflows
3
.
Source: AP NEWS
Maria Flynn, CEO of Jobs for the Future, advocates treating AI as a workflow augmentation tool rather than a complete substitute. "Think about AI as augmenting your workflow," Flynn advised, suggesting AI should function like an assistant for tasks such as email drafting or travel research, but not as a replacement for human oversight
1
.Flynn's experience with an in-house AI tool illustrates both the potential and pitfalls of workplace AI integration. When requesting discussion questions for a meeting, the AI initially provided contextually inappropriate suggestions but improved after feedback. However, the tool also confused completed work with funding proposals, demonstrating the critical need for human verification
2
.Related Stories
Expert Recommendations and Privacy Concerns
Legal experts emphasize the importance of verification despite AI's convincing output. Justin Daniels, an Atlanta-based attorney with Baker Donelson, warns against assumptions based on AI's plausible-sounding responses. "People are making an assumption because it sounds so plausible that it's right, and it's convenient," Daniels noted, stressing that checking citations and reviewing AI-summarized contracts remains essential despite being "inconvenient and time-consuming"
3
.Beyond accuracy concerns, AI tools raise significant privacy issues across industries. Workers must carefully consider what information they input into AI prompts to protect confidential employer and client data. Danielle Kays, a Chicago-based partner at Fisher Phillips, highlights additional concerns with AI note-taking tools, noting that many jurisdictions require participant consent for recording conversations and that some discussions should remain privileged and confidential
1
.References
Summarized by
Navi
[2]
[3]
Related Stories
Weekly Highlights
Weekly Highlights
Today's Top Stories
Your Daily Dose of Curated AI News
Don’t drown in AI news. We cut through the noise - filtering, ranking and summarizing the most important AI news, breakthroughs and research daily. Spend less time searching for the latest in AI and get straight to action.
