Databricks Launches Serverless Lakebase Database, Challenging Legacy Database Market

Following Neon acquisition, Databricks launches serverless Lakebase database - SiliconANGLE

Following Neon acquisition, Databricks launches serverless Lakebase database Databricks Inc. today introduced a new serverless database called Lakebase that can process more than 10,000 queries per second. The service is based on PostgreSQL, a popular open-source relational database. It's powered by technology that Databricks obtained through its recent acquisition of Neon Inc., a startup with an eponymous serverless PostgreSQL platform. The deal was reportedly worth about $1 billion. PostgreSQL was originally designed to store relational data organized into rows and columns. By installing an open-source extension called pgvector, developers can add support for vectors, the data structures in which artificial intelligence models keep information. Lakebase is compatible with pgvector as well as another extension, PostGIS, that adds support for geospatial data such as maps. Databricks has integrated Lakebase into its flagship data lakehouse platform. As a result, customers can manage both using a single set of controls. According to Databricks, the integration addresses one of the main challenges involved in running databases such as PostgreSQL: they historically had to be managed separately from a company's analytics environment. That arrangement created additional work for administrators. Databricks' platform ships with a data governance tool called Unity Catalog. It enables administrators to regulate which user can access what record, as well as monitor how business information is used. Customers can use Unity Catalog's controls to manage their Lakebase environments. Databricks' platform stores information in structures called lakehouse tables. According to the company, customers can automatically sync data between lakehouse tables managed with Unity Catalog and Lakebase records. There are three ways of going about the task. Databricks' platform can be configured to sync data on a one-off basis, automatically when certain conditions are met or continuously. The integration also extends to other areas. Customers may use the Databricks SQL Editor to run queries on Lakebase data. Meanwhile, custom applications deployed on the company's platform using the Databricks Apps tool can likewise access the new PostgreSQL service. Databricks manages the infrastructure on which Lakebase runs. According to the company, each customer database is spread across multiple cloud zones to mitigate the impact of outages. If one of the zones experiences downtime, the rest can still process queries. A recovery feature allows customers to restore records created up to 35 days ago in case user error or technical issues cause data loss. "It's now possible to launch thousands or even millions of agents with their own databases cost-effectively, without requiring specialized engineers (e.g. DBAs) to maintain/support staging environments; this reduces TCO," a group of Databricks executives wrote in a blog post. Lakebase is currently in public preview. Databricks plans to follow up today's launch by releasing "significant improvements" for the database in the coming months.

Databricks Looks To Disrupt Legacy Database Space With New 'Lakebase' Offering

Transactional database technology "hasn't actually changed that much in the last 40 years" and is inadequate for today's operational AI applications and agents, Databricks co-founder and CEO Ali Ghodsi said Wednesday during his keynote at the company's Data + AI Summit. Databricks has launched a fully managed Postgres database for building data-intensive applications and AI agents, throwing down a challenge to long-time database leaders such as Oracle and Microsoft. The new Databricks Lakebase offering, unveiled Wednesday during CEO and co-founder Ali Ghodsi's keynote address at the Databricks Data + AI Summit in San Francisco, adds an operational database layer to the Databricks Data Intelligence Platform that the company said meets the need for fast, reliable data by today's data applications, AI agents, recommendation engines and automated workflows. In launching Lakebase, Ghodsi took aim at "traditional transaction" databases such as the Oracle Database, Microsoft SQL Server, the MySQL database (also owned by Oracle) and others. Databricks, in the Lakebase announcement, said operational OLTP (online transaction processing) databases underpin every application and today are a $100-billion-plus market. [Related: If You Build Them: Databricks To Launch New Workflow, AI Agent Development Tools] In his keynote, Ghodsi said Lakebase and other newly announced Databricks products are a way to "bring data and AI to as many people on the planet as possible. The reality on the ground is that it's still really hard to succeed with data and AI." He noted that data today is spread across cloud and on-premises systems, is composed of both structured and unstructured data, is managed using customized data ETL (extract, transform and load) tools, and analyzed using multiple business intelligence tools within an organization. "The data estate is fragmented. This creates complexity within most organizations," Ghodsi said. "This is why everything moves slowly within organizations." For the Lakebase unveiling, legacy transaction databases were the focus of the CEO's speech. "If you look at these databases, they were really built for a different era. Technology in transactional databases hasn't actually changed that much in the last 40 years," Ghodsi said. The CEO said a major reason for this is that "data is so sticky" and once an organization has implemented a transactional database and loaded data into it, "it's just nearly impossible" to move off of it or move data to a new database. "And once you lock yourself into that, those vendors don't need to really innovate. They don't need to challenge the status quo because you can't go anywhere," Ghodsi said. Along with hindering analytics and AI, legacy databases are also a roadblock to developing AI capabilities into today's operational applications, he said. "If you look at all the different categories of software that exists today, whether it's CRM, ERP, human capital management, contracting and so on and so forth, it doesn't require that much imagination to see how AI is going to completely disrupt each one of these. AI is going to be infused into these." Databricks executives also make the argument that operational and analytical systems need to converge to reduce latency between AI systems and to provide enterprises with current information for making real-time decisions. While analytics systems have evolved, "OLTP databases are kinda stuck in the past," said Reynold Xin, Databricks co-founder and chief architect, joining Ghodsi on stage. They are slow, difficult to provision and difficult to scale, "and so are fairly disconnected from modern day developer workflows," he said. Lakebase, currently in public preview, is being used by some 300 Databricks customers. It is based on the open-source Postgres database technology Databricks acquired through its $1-billion acquisition of database startup Neon. While that deal was just announced last month, Ghodsi said Databricks was an investor in Neon and had been working with the startup prior to the acquisition. Lakebase incorporates a data lakehouse architecture that Databricks says is more flexible that legacy databases and separates compute and storage for independent scaling. It's cloud-native architecture reduces latency and supports high concurrency and high availability needs, according to the company. It automatically synchs data to and from lakehouse tables and is integrated with Databricks Apps development capabilities and Unity Catalog. Databricks also says Lakebase is designed for AI and agent development with its "unique branching capability" that enables low-risk development for both programmers and agent-based development. "We actually feel like this is how databases should be built in the future," Xin said. "And our prediction is that every other database, every other transactional database, will evolve towards this architecture in the coming years." "We think this is going to be the future of all databases," Ghodsi said. During a press conference following the keynote, a CRN reporter asked Ghodsi how he thought Lakebase would rate with Databricks channel partners given that some work with other database vendors and have invested in their database technologies. "For this era, you need a different type of database," Ghodsi said, maintaining that many businesses and organizations are "eager to get off of those old databases. I think we're going to see many more of those projects." And that, he said, will grow an ecosystem of VARs, systems integrators and consulting partners to help businesses and organizations undertake those data migration projects. "I think it's a massive opportunity," Ghodsi said. He noted that database migration projects, especially transactional databases, are traditionally very difficult but that data migrations from OLTP databases to Lakebase will be easier. He added that partners will also benefit from the elimination of what he called vendor lock-in. But he added that this transition won't happen overnight. "I think this is going to be a long marathon and we're taking the first step on this marathon." Databrick's overall goal of improving data pipelines and data quality for data analytics, AI and generative AI tasks resonates with Hugh Burgin, principal of the AI and data consulting practice at IT services giant EY, a Databricks Global Elite level partner that develops solutions and accelerators on the Databricks platform. In an interview with CRN Burgin said technologies like those offered by Databricks to improve data management and data quality is a boon for EY when it undertakes data analytics and AI projects for clients. "The biggest value, the biggest ROI opportunity, is to access that AI-ready data across the enterprise," he said. Burgin said EY takes the position that between 80 and 90 percent of the dollars spent on AI projects should be spent on the underlying data infrastructure. "It's a big opportunity for us," he said. In addition to the Lakebase announcement, Databricks, as expected, also unveiled Agent Bricks, a new unified workspace for building high-quality AI agents. Some of Wednesday's nearly three-hour keynote session was devoted to detailing and demonstrating the capabilities of Agent Bricks, which is currently in beta. "These are production agents [that] auto-optimize on your data," Ghodsi said onstage. Agent Bricks addresses several common problems around AI agents today including difficulty assessing their performance, choosing which techniques to best optimize agents, and balancing agent quality against the costs of developing agents and putting them into production, the CEO said. "We think that with Agent Bricks, it is both easy to use [and] also able to produce incredibly high-quality agents," said Hanlin Wang, Databricks CTO for Neural Networks, also speaking on stage during the keynote. Databricks also announced that Databricks Apps, initially unveiled in October 2024, is now generally available. Databricks Apps, according to the company, makes it easy to build secure and governed data intelligence applications deployed directly in Databricks and governed through the company's Unity Catalog data governance software. Ghodsi said that even before being generally available, Databricks Apps has been adopted by 2,500 customers who used it to develop more than 20,000 applications on the Databricks platform. The company also unveiled a free edition of the Databricks Data Intelligence Platform targeting universities, as well as individuals including students, hobbyists and "aspiring professionals." Databricks also pledged a $100 million investment in global data and AI education which in a press release the company said is aimed at "closing the industry-wide talent gap and preparing the next generation of data and AI engineers, data analysts and data scientists." Databricks also announced that it is open sourcing its core declarative ETL framework as Apache Spark Declarative Pipelines. That follows the recent launch of Apache Spark. 4.0.