Supreme Court's AI Disclosure Rule Faces Judicial Pushback as Fabricated Cases Emerge

Delhi HC Judge Questions Supreme Court's AI Disclosure Rule

We missed this earlier: Delhi High Court Justice Prathiba M Singh has questioned the Supreme Court's proposal to make lawyers disclose their use of artificial intelligence in court filings. She argued that the requirement could increase compliance work without necessarily improving accountability, according to a Bar & Bench report. Speaking at the International ADR Conference 2026, Singh joined a discussion with Senior Advocate J Sai Deepak. She said lawyers already deal with several technology-related requirements. These include requirements relating to OCR and electronic filing. "I think disclosure norms is just going to make lawyers' life more difficult." She said mandatory declarations could eventually become routine paperwork rather than a meaningful safeguard. "They have to OCR documents, e-file documents and at the end of the day, even if you ask for disclosures, they all become templated affidavits, right?" Singh also pointed to India's adversarial legal system as one check against inaccurate AI-generated material. If one side relies on fabricated or incorrect material, she said, opposing counsel can challenge it before the court. At the same time, she did not argue against lawyers using AI. Instead, she stressed that people must verify AI-generated material before relying on it. "AI is going to be used. You wouldn't want lawyers to come and lie to you saying they don't use AI. They should use AI, but with the human verification standard. Without human verification, it can be an absolute disaster." She referred to a GST matter involving an order that cited four judgments. Three of those judgments did not exist. The Delhi High Court then set aside the order. She also recalled an appeal involving a trial court judgment. Its reasoning did not match the pleadings. The High Court observed, prima facie, that the judgment appeared to have been drafted using AI. Singh said AI could help judges and lawyers with research, summaries and drafting, but should not replace human decision-making. Judiciary-Controlled AI: She also called for the judiciary to build its own AI platform instead of relying on public tools for sensitive court material. "We could use a large language model, but we should get our own AI platform prepared, with our own algorithms, the data residing within the control of the judiciary, the models being available within the judiciary." Her concern was particularly acute for matrimonial and criminal matters, where court records may contain sensitive personal information. Singh was more cautious about deploying AI in district courts. She said the lower judiciary first needs stronger digital and physical infrastructure and suggested that this process could take another five to seven years. She instead highlighted cheaper live-transcription systems as a more immediate intervention. "Get a cheaper live transcription software for the entire country, and you will see how criminal trials will move." Why the Disclosure Debate Matters: Singh's remarks come as the Supreme Court considers its draft Regulations for Use of Artificial Intelligence in Courts, 2026, released on June 3. The draft allows AI for activities such as legal research, transcription, translation, filing scrutiny and case management, but bars AI-only adjudication, risk scoring and certain forms of profiling. It also requires human oversight and verification for several uses. For lawyers, the most contested provision is the requirement to disclose the "AI-assisted character" of documents prepared using AI. The Supreme Court Advocates-on-Record Association, or SCAORA, opposed that requirement in July, calling it "unworkable". It argued that lawyers are already professionally responsible for the accuracy of their filings. It proposed a narrower certification instead. Advocates would only need to confirm that they had personally verified citations and legal authorities. The dispute comes amid repeated cases of fake or fabricated AI-generated citations reaching Indian courts. In July, the Supreme Court adopted a zero-tolerance position on hallucinated precedents. The move followed cases in which tribunals relied on non-existent judgments or material containing fabricated citations. Lawyers, Courts and Tech Firms Push Back Differently: The Court has nevertheless been expanding AI use for judicial administration and research. Its 2025 white paper supported AI for tasks such as research, transcription, translation and filing scrutiny. It also insisted that judges remain responsible for final decisions. The paper said humans must verify AI output. Industry concerns are different: In its comments on the draft rules, Nasscom largely supported the framework. It asked the Supreme Court to define terms such as "high-risk applications." Nasscom also sought clarity on technical audits. It said these audits should not automatically require companies to disclose source code or model architecture. The three positions expose the main tension in the Supreme Court's proposed AI framework. Courts want safeguards against hallucinations, privacy breaches and opaque automated systems. Lawyers are resisting blanket disclosure requirements. Technology companies also want clearer limits on audits and access to proprietary systems.

Artificial Intelligence, Judicial Integrity, and The Rule Of Law

In a landmark ruling, the Supreme Court of India addressed the growing threat of AI-generated fabricated legal precedents in judicial proceedings. The judgment in Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd. & Anr. establishes a zero-tolerance policy toward hallucinated case law and sets guidelines for the responsible use of AI in the legal profession. Civil Appeal No. 11950 of 2025 | Supreme Court of India | 2 July 2026 | Justice Pamidighantam Sri Narasimha and Justice Alok Aradhe Introduction: AI in the Legal Profession The increasing integration of Artificial Intelligence (AI) into legal practice has revolutionised the way legal professionals conduct research, analyse precedents, and prepare legal submissions. AI-powered legal tools have significantly enhanced efficiency by enabling lawyers and judges to process vast amounts of legal information within a short period. However, this technological advancement has also introduced a serious challenge in the form of AI "hallucinations" -- instances where generative AI systems produce fabricated judicial precedents, inaccurate legal propositions, or fictitious quotations while presenting them as authentic. Such inaccuracies pose a unique threat to the administration of justice because judicial decision-making is fundamentally dependent upon the authenticity and reliability of legal authorities. Recognising both the opportunities and risks associated with Artificial Intelligence, the Supreme Court of India, in Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd. & Anr., delivered a judgment addressing the responsible use of AI within the judicial system. Although the appeal originated from proceedings under the Insolvency and Bankruptcy Code, 2016, the Court treated the matter as one involving a broader institutional concern. It observed that technological innovation has consistently strengthened judicial administration through initiatives such as: E-filing* Digitisation of court records * Virtual hearings * However, generative AI differs fundamentally from these conventional technological tools because it generates legal content rather than merely retrieving existing information. Consequently, unchecked dependence on AI without adequate human verification has the potential to compromise judicial integrity and public confidence in the legal system. * Facts of the Case * Insolvency Proceedings and Tribunal Orders * The dispute arose from insolvency proceedings initiated by Jammu and Kashmir Bank Limited under Section 7 of the Insolvency and Bankruptcy Code, 2016, against Essel Infraprojects Limited (EIL). EIL had executed a corporate guarantee in favour of the Bank for credit facilities extended to Pan India Utilities Distribution Company Limited (PIUDCL). * Following defaults in repayment, the National Company Law Tribunal (NCLT), Mumbai, admitted the application, appointed an Interim Resolution Professional, and declared a moratorium under Section 14 of the Code. * The appellant challenged this order before the National Company Law Appellate Tribunal (NCLAT), arguing that subsequent corporate restructuring through demerger and amalgamation, together with a renewed sanction letter, had extinguished the corporate guarantee. The NCLAT rejected these submissions and upheld the admission order passed by the NCLT. * Discovery of Fabricated Precedents * The appellant's counsel pointed out that several judicial precedents relied upon by the NCLT were either entirely non-existent or genuine judgments accompanied by fabricated extracts that did not exist in the reported decisions. * Upon conducting an independent examination, the Supreme Court confirmed that multiple authorities cited by the Tribunal were indeed fictitious or incorrectly attributed. The Court concluded that reliance upon such hallucinated material had fundamentally tainted the judicial process, rendering the decisions of both the NCLT and the NCLAT legally unsustainable. * Consequently, it set aside the impugned orders and restored the Section 7 application to the NCLT for fresh adjudication in accordance with law, without expressing any opinion on the merits of the insolvency dispute itself. * Legal Framework * Although the case was filed under Section 7 of the Insolvency and Bankruptcy Code, 2016, the Supreme Court went beyond the insolvency issues and focused on a larger question -- whether a court decision based on fake legal authorities can be valid. * The Court explained that every judicial decision must be based on genuine laws and authentic court judgments. If a court relies on false or non-existent precedents, it affects the fairness of the judicial process and weakens public trust in the justice system. * The Court also made it clear that it is not against the use of Artificial Intelligence in the legal profession. It accepted that AI could help lawyers and judges by making legal research faster and more efficient. However, AI should only be used as a support tool. Every case, legal principle, or judgment generated by AI must be checked and verified by a human before it is used in court. * Therefore, the judgment creates a balance between encouraging technology and protecting the integrity of the justice system. * The Court's Reasoning * AI as a Content Generator, Not a Research Tool * The Supreme Court observed that Artificial Intelligence is different from ordinary research tools because it can create legal content instead of simply finding existing information. While this makes legal work quicker, AI can also generate false case laws, fake quotations, and incorrect legal principles. * The Court warned that if judges and lawyers start depending on AI without checking its accuracy, it could seriously affect the quality of judicial decision-making. * Hallucinated Precedents as a Hidden Danger * The Court described AI-generated fake legal material as a hidden danger to the justice system. It compared such hallucinations to the release of methyl isocyanate gas, explaining that they are difficult to detect but can cause serious damage to the judicial process. * According to the Court, using fake precedents does not merely result in a legal mistake; it damages the credibility of the entire justice system because judicial decisions must always be based on real and verified legal authorities. * Findings on the NCLT and NCLAT Orders * After examining the records, the Supreme Court found that the NCLT had relied on several judgments that either did not exist or contained paragraphs that were never part of the original judgments. These mistakes were not noticed by the NCLAT while hearing the appeal. * Since both forums relied on incorrect legal material, the Supreme Court held that their decisions could not be allowed to stand. * Zero-Tolerance Policy and Institutional Directions * The Court therefore adopted a policy of zero tolerance towards AI-generated precedents that are used without verification. It held that: