Wall Street Giants Warn of New AI Risks: Hallucinations, Cybercrime, and Employee Morale

Wall Street Flags New AI Risks From Hallucinations, Criminal Use

Goldman Sachs Group Inc., Citigroup Inc., JPMorgan Chase & Co. and other Wall Street firms are warning investors about new risks from the increasing use of artificial intelligence, including software hallucinations, employee-morale issues, use by cybercriminals and the impact of changing laws globally. The dangers newly flagged in the bank's annual reports include flawed or unreliable AI models, increased competition and new regulations restricting use of AI. JPMorgan, for example, said AI could cause "workforce displacement" that might affect staff morale and retention, and increase competition for hiring employees with the necessary technological skills, according to the firm's 2024 10-K.

Banking Giants Warn AI Can Bolster Cybercrime and Lower Morale | PYMNTS.com

Wall Street banking giants have reportedly begun warning investors about risks stemming from AI use. As Bloomberg News reported Wednesday (March 12), those risks include so-called artificial intelligence (AI) "hallucinations," use of the technology by cybercriminals and its effect on employee morale. For example, the report said, JPMorgan said in a recent regulatory filing that AI could bring about "workforce displacement" that could affect worker morale and retention, while increasing competition for employees with the appropriate tech background. Bloomberg notes that while banks have in recent years been pointing to AI risks in their annual reports, new concerns are emerging as the financial world embraces the technology. It's a balancing act: keeping on top of the latest AI advancements to retain customers, while also dealing with the threat of cybercrime. "Having those right governing mechanisms in place to ensure that AI is being deployed in a way that's safe, fair and secure -- that simply cannot be overlooked," Ben Shorten, finance, risk and compliance lead for banking and capital markets in North America at Accenture, said in an interview. "This is not a plug-and-play technology." The Bloomberg report adds that banks are at risk of using technologies that may be built on outdated, biased or inaccurate financial data sets. Citigroup said that as it introduces generative AI at its company, it faces risks of analysts working with "ineffective, inadequate or faulty" results produced. This data could also be incomplete, biased or inaccurate, which "could negatively impact its reputation, customers, clients, businesses or results of operations and financial condition," the bank said in its 2024 annual report. PYMNTS wrote recently about the use of AI in cybercrime, arguing that it helped add to a larger landscape of cyberattacks in 2024 that included ransomware, zero-day exploits and supply chain attacks. "It is essentially an adversarial game; criminals are out to make money and the [business] community needs to curtail that activity. What's different now is that both sides are armed with some really impressive technology," Michael Shearer, chief solutions officer at Hawk, said in an interview with PYMNTS. And last month, PYMNTS examined efforts by Amazon Web Services (AWS) to combat AI hallucinations using automated reasoning -- a method rooted in centuries-old principles of logic. The technique is a major leap in making AI outputs more reliable, which is particularly valuable for heavily regulated industries such finance and health care, AWS Director of Product Management Mike Miller said in an interview.

Wall Street warns of new AI risks from hallucinations, criminal use

Goldman Sachs, Citigroup, JPMorgan Chase and other Wall Street firms are warning investors about new risks from the increasing use of artificial intelligence, including software hallucinations, employee-morale issues, use by cybercriminals and the impact of changing laws globally. The dangers newly flagged in the bank's annual reports include flawed or unreliable AI models, increased competition and new regulations restricting use of AI. JPMorgan, for example, said AI could cause "workforce displacement" that might affect staff morale and retention, and increase competition for hiring employees with the necessary technological skills, according to the firm's 2024 10-K. Banks have been acknowledging AI-related risks in their annual reports for the past couple of years, but new concerns are cropping up as the financial sector increasingly embraces AI via their own software or third-party offerings. If banks don't keep up to date with the latest AI developments, they risk losing customers and business, they said in their annual reports. But increased AI use also opens them up to risks from cyberattacks and misuse. "Having those right governing mechanisms in place to ensure that AI is being deployed in a way that's safe, fair and secure -- that simply cannot be overlooked," Ben Shorten, Accenture's lead for finance, risk and compliance for banking and capital markets in North America, said in an interview. "This is not a plug-and-play technology." Banks are at risk of piloting technologies that may be built using outdated, biased or inaccurate financial data sets. JPMorgan's annual report said there are dangers around developing and maintaining models that have the highest level of "data quality." Citigroup said that as it rolls out generative AI at select parts of the bank, there are risks of "ineffective, inadequate or faulty" results produced for its analysts. The data could also be incomplete, biased or inaccurate, which "could negatively impact its reputation, customers, clients, businesses or results of operations and financial condition," according to its 2024 annual report. Integrating AI Goldman Sachs said that while it's increased its investment in digital assets, blockchain and AI, growing competition poses risks to integrating AI technologies in a timely enough manner to boost productivity, reduce costs and give clients better transactions, products and services, according to the firm's latest annual report. That could affect customer attraction and retention, Goldman said. Financial firms also run the risk of maintaining data privacy and regulatory compliance in an environment that is "less certain and rapidly evolving," Shorten said. In 2024, the EU Artificial Intelligence Act went into effect, establishing new rules on the use of AI systems in the region, where many U.S. banks have operations. "This act establishes rules for placing on the market, putting into service and using a lot of artificial intelligence systems in the EU," Shorten said. "The outlook for the U.S. and the U.S. market is less clear." Banks are using a combination of their own AI tools and ones acquired from outside providers. Citigroup is rolling out a suite of tools that can synthesize key information from public filings. AI @ Morgan Stanley Debrief is taking on rote tasks with a ChatGPT-like interface. And Goldman's private-wealth division is using AI to evaluate portfolios and analyze dozens of underlying positions, said Chief Information Officer Marco Argenti. "It's so important to take a responsible approach and really be applying controls so that you protect yourself from potential inaccuracies and hallucinations," he said last week at the Bloomberg Invest conference in New York. JPMorgan CEO Jamie Dimon said AI may be the biggest issue his bank is grappling with. In his annual shareholder letter, he likened AI's potential impact to that of the steam engine and said the technology could "augment virtually every job." Representatives for the banks declined to comment beyond the AI disclosures in their annual reports. As banks increasingly turn to AI, cybercriminals are doing the same -- and are becoming increasingly sophisticated in its use, according to Shorten. Accenture's most recent global survey of 600 cybersecurity executives in the banking industry found their teams are struggling to keep up with their organizations' AI adoption efforts. Among respondents, 80% believe generative AI is empowering criminals faster than banks can respond. Morgan Stanley said in its latest annual report that generative AI, remote work and integrating third-party technology could pose a risk to data privacy. The risks introduced by using AI while working from home will require firms to set up rules to avoid problems, Shorten said. "These steps are only going to increase in criticality," he said, "as attackers are being enabled by this technology faster than the banks are able to respond."