AI Hallucinations: Lessons for Companies and Healthcare

Council Post: What AI Hallucinations Can Teach Companies

Debanjan Saha is CEO of DataRobot and a visionary technologist with leadership experience at top tech companies such as Google, AWS and IBM. When using generative AI (GenAI) for marketing, advertising or entertainment, it might be acceptable to have the occasional response that is professionally written but factually inaccurate. For the large majority of GenAI use cases, however, the stakes are higher. This lack of confidence in GenAI outputs is holding leaders back from using it in high-stakes external interactions such as healthcare and finance. "Hallucinations" are just one of many challenges preventing teams from implementing GenAI: If you need to spot-check and research replies to ensure accuracy, you might just as well have done the job yourself. While unexpected outputs can range from merely annoying and counterproductive to potentially dangerous, AI hallucinations -- believe it or not -- might be useful in revealing how modernized and enterprise-grade your AI processes, checkpoints and management might be. First, let's take a closer look at the root causes of LLM hallucinations. 1. Training Data Limitations: LLMs are trained on vast datasets consisting of text from the internet, books and more. These datasets might contain inaccuracies, biases or outdated information. The LLM can learn and replicate these flaws in its outputs. 2. Interpretation And Inference Errors: LLMs generate responses based on patterns and associations in the data they have been trained on. They can misinterpret a query or make incorrect inferences, leading to responses that are factually incorrect or nonsensical. 3. Lack Of World Knowledge: While LLMs can simulate understanding through the vast amount of information they're trained on, they don't possess true understanding or awareness. This can result in errors when the model attempts to generate information about new events, complex concepts or specific expertise areas outside its training data. 4. Overgeneralization: LLMs might overgeneralize from the training data, leading to responses that seem plausible but are incorrect or based on factually inaccurate training material. 5. Context Limitations: LLMs can struggle with maintaining and understanding context over longer conversations or texts. This might lead to responses that are inconsistent or not fully aligned with the initial query or the ongoing conversation. 6. Model Complexity And Opacity: The internal workings of LLMs are complex and not fully understood, even by their creators. This complexity can lead to unexpected behaviors, including hallucinations that are difficult to predict or explain. AI hallucinations highlight gaps in AI's build, governance and operation processes. CIOs and AI leaders need to examine each of these three critical areas with an eye toward reliability, stability and intervention to ensure that the outputs align with expected results. To do so, AI leaders must approach hallucinations as an integral part of the AI development lifecycle. AI hallucinations let CIOs and AI leaders know where they need to invest to create state-of-the-art processes that are built to handle GenAI. AI leaders, therefore, require real-time monitoring, logging and observability of GenAI outputs to detect anomalies. They also must create feedback loops for users and obtain expert reports on inaccuracies as well as hypergranular lineage of the prompt and the generated response to see where they need to augment the LLMs' understanding of a topic. As with all new tech, the focus and excitement is on building new GenAI applications. Still, the real value of GenAI will only be captured once CIOs and AI leaders can feel confident in the outputs. This confidence requires that AI leaders focus not just on building, but also on lifecycle management, maintenance, oversight, governance and security to facilitate the early identification of potential issues. More importantly, leaders must ensure the continuous refinement of GenAI models through iteration and intervention. That said, hallucinations may be much more than simple errors that need to be fixed. They may offer alternate approaches to problem-solving and creativity. Perhaps the most intriguing aspect of AI hallucination is its potential to enhance and even provide a proxy for human creativity. Sophisticated hallucinations often involve GenAI providing unexpected combinations of ideas or reconfiguring patterns in ways not explicitly present in its training data. This type of "mistake" is similar to the underpinnings of human creativity. "Human" creativity involves activating diverse and often distant brain networks to recombine large amounts of information in novel ways. AI, particularly in its use of neural networks, mimics this process by drawing on vast datasets to produce new patterns or ideas not present in its training data. The resulting AI hallucinations are like the brain's creative leaps in connecting disparate ideas. These leaps hint at early-stage creativity within the GenAI, offering exploration, learning and problem-solving reminiscent of human imagination. First things first: CIOs and AI leaders need to have confidence in GenAI before they begin to use it to create solutions to complex problems. It will take time and experience to recognize when GenAI "creativity" should be accepted, encouraged or reined in. Users will also have to be very clear on whether their GenAI is demonstrating "synthetic creativity" rather than offering factual outputs. CIOs and AI leaders will need to partner with users to ensure that GenAI isn't offering creativity when trusted outputs are what's called for. To achieve the balance between creativity and confidence, CIOs and AI leaders must ensure that building, governance and operation are completely seamless and unified in the AI lifecycle. Fractured infrastructure will only leave you with fractured visibility and a lack of confidence in how your AI initiatives are performing. Investing in streamlining your AI lifecycle is the first crucial step that allows organizations to use GenAI with confidence in higher-stakes interactions. Hallucinations hint at a huge and exciting opportunity for GenAI to provide "synthetic creativity" to solve problems in ways that are different, new and innovative. To do this, AI leaders need to embrace the current challenges around confidence and use errors to understand what areas need to be improved. Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?

Hallucinations And Constant Learning: Healthcare AI is Just Getting Started

A new study published in Nature undertook a systematic analysis of the ethical landscape involving the use and application of large language models in medicine and healthcare. The study found that while LLMs have numerous benefits with regard to data analytics, insight-driven decision making support and information accessibility, issues of fairness, bias and misinformation are still of key concern in the context of healthcare. Indeed, artificial intelligence technology and the use of LLMs in healthcare contexts have grown exponentially, especially with how rapidly the technology has developed over the last two years. Although the launch of Chat GPT catalyzed much of this work, the reality is that the research surrounding LLMs and the general incorporation of AI into industry use-cases has been prevalent for decades. Technology pundits, privacy stalwarts and industry leaders have issued concern with how rapidly this work is advancing -- growth which regulatory bodies simply have not been able to keep up with. Thus, organizations and leaders alike are attempting to develop frameworks to guide the development and ethical nuances for industry use-cases. Take for example the Coalition for Health AI, also popularly known as CHAI, which aims "to develop 'guidelines and guardrails' to drive high-quality health care by promoting the adoption of credible, fair and transparent health AI systems." Another example is the Trustworthy & Responsible AI Network (TRAIN), spearheaded by Microsoft and European organizations to operationalize ethical AI principles and create a network where best practices regarding the technology can be shared. The vast amount of investment and resources being placed on initiatives like these indicate just how important this agenda has become. The reason for this emphasis is well-founded, especially in the context of healthcare use-cases. AI in healthcare unlocks significant potential to ease workflows, help with insight driven decision-making, promote new methods of interoperability and even make the use of resources and time more efficient. However, in the larger timeline, the work surrounding these applications is still relatively nascent. Furthermore, with regards to data fidelity, LLMs are often deemed to be only as effective as the datasets and algorithms they are trained with. Therefore, innovators have to constantly ensure that the training data and methods that are being used are of the highest quality. Additionally, the data has to be relevant, updated, bias-free and backed by legitimate references, so that systems can continue to learn as paradigms evolve and new data emerges. Even with pristine training conditions and all of these criteria met, AI systems may still frequently produce hallucinations, or the generation of content that is confidently asserted as true, but is often inaccurate. To an end-user that does not have a better source of truth, these hallucinations can prove to be detrimental -- and in the context of healthcare, can become a significant concern. Therefore, the increasing focus on ethical AI and the development of guidelines for AI are crucial aspects of cultivating this revolutionary technology, and will ultimately be paramount to truly unlock its potential and value in a safe and sustainable manner.