AI Infrastructure and Data Cleansing: Key Challenges in the AI Era

Insights into AI infrastructure demands and CDO evolution - SiliconANGLE

Three insights you might have missed from theCUBE's coverage of the CDOIQ Symposium The artificial intelligence revolution is changing industry understandings of what a chief data officer is. AI infrastructure demands have driven the evolution of the role beyond the merging of the chief data and analytics officer positions. Unpacking the role of the CDO in the age of AI infrastructure demands was a central focus of last week's CDOIQ Symposium. Things are getting increasingly messy, according to theCUBE Research Chief Analyst Dave Vellante (pictured). "[There are] governance challenges, the AI confusion that's coming into the marketplace, the risks associated with that," Vellante said. "It's an increasingly important role, but a harder one." Analysts for theCUBE, SiliconANGLE Media's livestreaming studio, spoke with top executives and industry experts during the event. They explored the opportunities and challenges of the data revolution and what's enabling the data revolution to meet AI infrastructure demands. Here are three key insights you may have missed from the CDOIQ Symposium: Looking back in time to the inception of the CDOIQ Symposium, there were always questions around the specifics of a CDO's job. Those questions persist, according to theCUBE Research's Paul Gillin. "There's still a lot of ambiguity about what the CDO role is. Whether it shouldn't involve analytics and AI. Who's in charge of AI?" Gillin asked. "I would expect that after a decade, these issues would be worked out, but they don't seem to be resolved." Considering AI infrastructure demands, the question, then, is where to go from here. That will continue to be an important question as things grow more and more complex, according to Vellante. "I think that organizations are going to have real challenges trying to get their entire data estate together," Vellante said. "We had lived by the mantra of a decentralized organization back when we were in our IDG days. You can almost see it's very inefficient in a way ... tons of duplication. But maybe it's more effective to just live with that." When considering challenges and solutions in data management, applying product management discipline is very important, according to theCUBE Research's Sanjeev Mohan. It's also important to apply software development lifecycle best practices, or what is known as DevOps. "When you have all these best practices applied to data, data becomes a lot more compelling," Mohan said. Amid the confusion, it's important for those in the C-suite to collaborate effectively. Instead of working in isolation, the CDO must work closely with the chief information security officer, chief technology officer and chief information officer, according to Mohan. "People are confused, 'Who should I go talk to?' The businesses still have [the] same problem. Their problem hasn't changed," he said. "What is their problem? How do I get access to data as fast as it's produced in a trustworthy manner? That has not changed. Instead, we have created a mass of different organizations and roles and processes. I think this confusion needs to be tamed at some point." As the chief data officer role continues to evolve, it's becoming increasingly clear that those in the role from the back office to the boardroom must lead the charge in data-driven transformation. Data is more important than ever and has been recognized as a critical asset, according to Richard Wang, founder and director of CDO Education Inc. and founder and general chair of the annual CDOIQ Symposium. "When you're going to make that much investment, you better get good results," he said. "Data quality has bubbled up to be the blocker in this space, but it's not just data quality of yesteryears. It is data quality on unstructured data as well." There's a recognition that unstructured data poses new challenges. But it's important to ensure that it is grounded in quality and utility, according to Wang. "This is a reason why data quality has become a complex topic now," he said. "In structured data, we can say is it duplicate data, is it missing data, is a format correct, all of [those] metrics. But for unstructured, the metrics are undefined." Through it all, the CDO must be equipped to manage both governance and business growth amid an evolving landscape of AI. Culture is critical to keep in mind, and it's not necessarily integral for an AI leader to also be a technical leader, according to Mario Faria, Chief Data and AI Officer Program professor at Carnegie Mellon University. "The best chief data officers and CIOs that I have met in my life, they did not have a technical background. They were able to understand," he said. "They were able to put together an organization with skills that would help the company to achieve their results." Here's the complete video interview with Richard Wang: In this new era, companies and organizations have sought to respond to the data revolution and meet AI infrastructure demands. That includes a solution from Cloudera Inc., which has laid out its approach for what it calls a true hybrid model. "True hybrid ... is where you have harmony in a multiplicity of environments, where the environments are working cohesively together to allow you to democratize data effectively and to generate insights," said Shayde Christian, chief data and analytics officer at Cloudera. "I want to build a workload one time, and I want to run it anywhere in the world on one platform. We're the only true hybrid company, in my opinion." Solutions can also be pinpointed in other areas, including those provided by the federal government. Thanks to a funding boost, the Internal Revenue Service is looking to improve the IRS user experience for taxpayers with artificial intelligence. "We had more than 60 million taxpayers either call us for help or come into one of our taxpayer assistance centers," said Melanie Krause, chief operating officer of the IRS. "The volume of information that we were working with requires use of analytics, key use of data to drive decisions, because with that much work, one can't be successful without a strong technology, strong analytics focus." Microsoft Corp. is also seeking to contribute to this landscape, focusing on advanced data governance to streamline operations and enhance user experiences. In this new landscape, unified metadata models are emerging as a cornerstone. "We spent almost two years just building technology and infrastructure and the next three practicing it," said Karthik Ravindran, general manager of enterprise data at Microsoft. "We are hoping that with investments we're making in Purview, our customers can get faster the value from data, governing it responsibly and doing great things with their data versus having to build tech to do it." Strategic data management, enhanced by AI integration and strong governance, is also essential for businesses to solve specific problems. That's why incremental improvements and strong governance ensures continuous value delivery, according to Tom Godden, enterprise strategist and CxO advisor at Amazon Web Services Inc. "I'm seeing the customers be more successful that have a good governance program, that have good control over their data so they can enter into effective IP sharing types of agreements," he said. "But, again, it comes back to you need a good governance program in place." Here's the complete video interview with Shayde Christian:

AI data cleansing and challenges in data governance - SiliconANGLE

AI data cleansing takes center stage in evolving data governance landscape Anyone who has worked in the data industry for several decades can speak to its continued evolution. Today, the focus has shifted to artificial intelligence, emphasizing AI data cleansing and governance, as clean data is essential for effective AI implementation. That's an emphasis that many have been pushing forward in this new era. Data must be cleansed for analytics to be more reliable, according to Nusrath Mohammed (pictured), data practice leader at Tata Consultancy Services. "Now, because of AI, everybody wants to jump on that bandwagon. But then we realize data is not cleansed and then we have to go backwards and start cleaning that first," Mohammed said. "Now, we can jump on that bandwagon." Mohammed spoke with theCUBE Research's Dave Vellante at the CDOIQ Symposium, during an exclusive broadcast on theCUBE, SiliconANGLE Media's livestreaming studio. They discussed the importance of AI data cleansing and the evolving challenges in data governance. There are a lot of tools available in the generative AI world right now. But certain tools, such as Microsoft Corp.'s Purview, still pose plenty of questions, according to Mohammed. "It's a beautiful tool. But then I'm asking the question, that tool can be utilized only at the top layer, once we have the data lake has been built," she said. "It's at the top layer. But what about at the layer of the data inception?" When one investigates more about how generative AI can be utilized, the answer is that one must implement the technology at all levels. That clicked, according to Mohammed. "During the data inception, when your data is coming into your source systems, then apply gen AI there," she said. "Detect and give us a report up there and say, 'OK, your addresses are not matching their accurate addresses.'" It could even be implemented when a user is creating data, as help arrives to inform that user that they may have entered an address incorrectly. Then, what's taking place at a change level is also taking place at an interface level, according to Mohammed. "You bring it at the top level. Now, you want to bring your insights," she said. "You're going to have much more better outputs too ... this is how I see gen AI helping and supporting what we are doing." Advances in technology doesn't mean that embedded generative AI is set to solve every problem that a company faces. Machine learning skills, data science and other skills are all still required after generative AI has provided feedback, according to Mohammed. "Maybe I still need to apply the traditional tools to get to an actual solution. Those are some of the challenges or the talk which I'm not hearing and which I would like to hear," she said. "It's like, 'This is not my silver bullet. This is at least aiding me.'" AI copilots might one day be equipped to solve very difficult problems. But for now, copilots still have to learn what humans are doing, according to Mohammed. "The copilots might help, but not right now," she said. "It will learn from me what I'm doing on a day-to-day basis, and then it'll give me much more intelligent advice at the end. But I don't think it is giving me intelligent advice at this moment." Here's the complete video interview, part of SiliconANGLE's and theCUBE Research's coverage of the CDOIQ Symposium: