Google Introduces DataGemma: A New Approach to Tackle AI Hallucinations

Google joins the war on AI hallucination with its massive Data Commons knowledge graph

AI researchers create intentionally toxic training models to keep LLMs wholesome Key Takeaways Large language models like GPT-3 can return incorrect responses due to hallucination. Google's DataGemma uses Data Commons to improve language model accuracy. DataGemma employs RIG and RAG strategies to combat AI lies. Large language models are critical to today's AI features, whether wrapped around other tools, or working simply as Turing-tested chatbots. As predictive models (and not intelligent beings), they can't reason, and aren't particularly tied to the truth. That leads to LLMs returning incorrect, practically made-up responses, called hallucination. It's one of the biggest obstacles to overcome in the quest for truly human-like language output. Related Google is already delivering on the Apple Intelligence promise Android takes the lead with AI 6 As a world leader in data harvesting and cataloging, Google can feed LLMs accurate data better than anybody. To that end, Google just unveiled a set of open models, called DataGemma, designed to improve LLMs' abilities to discern truth from fiction. When words that sound right turn out to be right Mimicking human reason more effectively than ever Source: Google Data Commons, Google's mind-bogglingly vast knowledge graph, is its latest weapon against pervasive LLM inaccuracy. A knowledge graph isn't just a list of facts, but an overarching network of data, including relationships between the many points, and how they interact. Leveraging interconnected facts with high-utility instructions is one effective way to refine language models. Now, Data Commons is that data, and DataGemma that set of algorithms. Another way to refine LLMs AI researchers create intentionally toxic training models to keep LLMs wholesome You were supposed to use your powers for good... oh, you did? DataGemma employs two main strategies to combat AI lies: Retrieval-Interleaved Generation and Retrieval-Augmented Generation. Using RIG, an LLM receives a prompt, generates a likely answer, then checks that answer against a database of verified facts and statistics. RIG partly restricts a model's decision-making to the probabilities it was originally trained on. In RAG, a language model first gathers relevant data from its assigned knowledge graph, and evaluates that dataset for an answer. This allows engineers to continually curate the model's available dataset, instead of relying on the intial training data. Google's recent blog post does an excellent job of making these as confusing as possible, but conceptually, they're not too complicated. What makes DataGemma so special? Mostly the fact that it uses Google's resources RIG and RAG aren't new, and various AI models use one or both in some ways. The difference is that DataGemma's operated by one of the biggest data hoarders ever. Google is only beginning the combined RIG and RAG approach to navigating the publicly available Data Commons graph. But it's committed to sharing its research to benefit the machine learning industry as a whole, offering phased-in access as the work progresses. No other group can compete with Google's combination of processing power, data storage and management, and engineering resources. So this massive undertaking could make a huge difference in LLM accuracy moving forward. Whether you think AI is humanity's savior or an overhyped customer service bot (it's actually somewhere in between), more truthful LLM responses can only be beneficial. Visit Google's DataGemma research post for more detailed explanations of RIG and RAG. Related 7 best ChatGPT alternatives Try these AI tools when ChatGPT is down

DataGemma: Google's open AI models mitigate hallucination on statistical queries

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More Google is expanding its AI model family while addressing some of the biggest issues in the domain. Today, the company debuted DataGemma, a pair of open-source, instruction-tuned models that take a step toward mitigating the challenge of hallucinations - the tendency of large language models (LLMs) to provide inaccurate answers - on queries revolving around statistical data. Available on Hugging Face for academic and research use, both new models build on the existing Gemma family of open models and use extensive real-world data from the Google-created Data Commons platform to ground their answers. The public platform provides an open knowledge graph with over 240 billion data points sourced from trusted organizations across economic, scientific, health and other sectors. The models use two distinct approaches to enhance their factual accuracy in response to user questions. Both methods proved fairly effective in tests covering a diverse set of queries. The answer to factual hallucinations LLMs have been the breakthrough in technology we all needed. Even though these models are just a few years old, they are already powering a range of applications, right from code generation to customer support, and saving enterprises precious time/resources. However, even after all the progress, the tendency of models to hallucinate while dealing with questions around numerical and statistical data or other timely facts continues to be a problem. "Researchers have identified several causes for these phenomena, including the fundamentally probabilistic nature of LLM generations and the lack of sufficient factual coverage in training data," Google researchers wrote in a paper published today. Even traditional grounding approaches have not been very effective for statistical queries as they cover a range of logic, arithmetic, or comparison operations. Public statistical data is distributed in a wide range of schemas and formats. It requires considerable background context to interpret correctly. To address these gaps, Google researchers tapped Data Commons, one of the largest unified repositories of normalized public statistical data, and used two distinct approaches to interface it with the Gemma family of language models -- essentially fine-tuning them into the new DataGemma models. The first approach, called Retrieval Interleaved Generation or RIG, enhances factual accuracy by comparing the original generation of the model with relevant stats stored in Data Commons. To do this, the fine-tuned LLM produces natural language queries describing the originally generated LLM value. Once the query is ready, a multi-model post-processing pipeline converts it into a structured data query and runs it to retrieve the relevant statistical answer from Data Commons and back or correct the LLM generation, with relevant citations. While RIG builds on a known Toolformer technique, the other approach, RAG, is the same retrieval augmented generation many companies already use to help models incorporate relevant information beyond their training data. In this case, the fine-tuned Gemma model uses the original statistical question to extract relevant variables and produce a natural language query for Data Commons. The query is then run against the database to fetch relevant stats/tables. Once the values are extracted, they, along with the original user query, are used to prompt a long-context LLM - in this case, Gemini 1.5 Pro - to generate the final answer with a high level of accuracy. Significant improvements in early tests When tested on a hand-produced set of 101 queries, DataGemma variants fined-tuned with RIG were able to improve the 5-17% factuality of baseline models to about 58%. With RAG, the results were a little less impressive - but still better than baseline models. DataGemma models were able to answer 24-29% of the queries with statistical responses from Data Commons. For most of these responses, the LLM was generally accurate with numbers (99%). However, it struggled to draw correct inferences from these numbers 6 to 20% of the time. That said, it is clear that both RIG and RAG can prove effective in improving the accuracy of models handling statistical queries, especially those tied to research and decision-making. They both have different strengths and weaknesses, with RIG being faster but less detailed (as it retrieves individual statistics and verifies them) and RAG providing more comprehensive data but being constrained by data availability and the need for large context-handling capabilities. Google hopes the public release of DataGemma with RIG and RAG will push further research into both approaches and open a way to build stronger, better-grounded models. "Our research is ongoing, and we're committed to refining these methodologies further as we scale up this work, subject it to rigorous testing, and ultimately integrate this enhanced functionality into both Gemma and Gemini models, initially through a phased, limited-access approach," the company said in a blog post today.

DataGemma: Using real-world data to address AI hallucinations

Large language models (LLMs) powering today's AI innovations are becoming increasingly sophisticated. These models can comb through vast amounts of text and generate summaries, suggest new creative directions and even draft code. However, as impressive as these capabilities are, LLMs sometimes confidently present information that is inaccurate. This phenomenon, known as "hallucination," is a key challenge in generative AI. Today we're sharing promising research advancements that tackle this challenge directly, helping reduce hallucination by anchoring LLMs in real-world statistical information. Alongside these research advancements, we are excited to announce DataGemma, the first open models designed to connect LLMs with extensive real-world data drawn from Google's Data Commons. Data Commons is a publicly available knowledge graph containing over 240 billion rich data points across hundreds of thousands of statistical variables. It sources this public information from trusted organizations like the United Nations (UN), the World Health Organization (WHO), Centers for Disease Control and Prevention (CDC) and Census Bureaus. Combining these datasets into one unified set of tools and AI models empowers policymakers, researchers and organizations seeking accurate insights. Think of Data Commons as a vast, constantly expanding database filled with reliable, public information on a wide range of topics, from health and economics to demographics and the environment, which you can interact with in your own words using our AI-powered natural language interface. For example, you can explore which countries in Africa have had the greatest increase in electricity access, how income correlates with diabetes in US counties or your own data-curious query. As generative AI adoption is increasing, we're aiming to ground those experiences by integrating Data Commons within Gemma, our family of lightweight, state-of-the art open models built from the same research and technology used to create the Gemini models. These DataGemma models are available to researchers and developers starting now. DataGemma will expand the capabilities of Gemma models by harnessing the knowledge of Data Commons to enhance LLM factuality and reasoning using two distinct approaches: 1. RIG (Retrieval-Interleaved Generation) enhances the capabilities of our language model, Gemma 2, by proactively querying trusted sources and fact-checking against information in Data Commons. When DataGemma is prompted to generate a response, the model is programmed to identify instances of statistical data and retrieve the answer from Data Commons. While the RIG methodology is not new, its specific application within the DataGemma framework is unique.