Databricks Unveils Synthetic Data API to Streamline AI Agent Evaluation

Databricks Introduces Synthetic Data API for AI Agent Evaluation

Databricks, a leader in the data ecosystem, has unveiled a new Application Programming Interface (API) designed to generate synthetic datasets for machine learning projects 1

1

. This innovative tool is integrated into the company's Mosaic AI Agent Evaluation platform, which is part of their flagship data lakehouse offering 1

1

2

2

.

The Need for Synthetic Data in AI Development

The introduction of this API addresses a significant challenge in AI development: the time-consuming and complex process of evaluating AI agent performance. By enabling the creation of high-quality artificial datasets, Databricks aims to streamline the development workflow, reducing the need for constant consultation with subject matter experts (SMEs) and accelerating the path to production for AI agents 2

2

.

How the Synthetic Data API Works

The process of creating a dataset with the new API involves three main steps:

1. Uploading a frame or file collection containing relevant business information.
2. Specifying the number of questions and answers to be generated.
3. Optionally providing additional instructions to customize the API's output 1

   1

   .

The API is designed to generate question and answer collections, which are particularly useful for developing applications powered by large language models 1

1

. Importantly, the synthetic answers produced are sets of facts required to answer the questions, rather than complete responses written by the language model. This approach facilitates faster review and editing by SMEs 1

1

.

Integration with Mosaic AI Agent Evaluation

The synthetic data capabilities are tightly integrated with Databricks' Mosaic AI Agent Evaluation platform. This integration allows developers to generate high-quality evaluation datasets for preliminary assessment quickly, reducing the workload on SMEs to final validation and accelerating the iterative development process 2

2

.

Performance Improvements and Future Enhancements

Internal tests conducted by Databricks have shown significant improvements in agent performance across various metrics when using the synthetic data for evaluation and improvement. For instance, they observed a nearly 2X increase in the agent's ability to find relevant documents and improvements in the overall correctness of responses 2

2

.

Looking ahead, Databricks plans to release several enhancements to the API in early 2024, including:

1. A new graphical interface for faster error checking of question-answer pairs.
2. Tools for tracking changes in synthetic datasets over time 1

   1

   .

Competitive Advantage

While there are other tools available for generating synthetic datasets, Databricks' offering stands out due to its seamless integration with the Mosaic AI Agent Evaluation platform. This integration eliminates the need for developers to leave their workflows, streamlining the entire process from data generation to agent evaluation 2

2

.

As enterprises increasingly adopt compound AI agents capable of reasoning and handling diverse tasks across different domains, Databricks' synthetic data API represents a significant step forward in simplifying the development and evaluation of these sophisticated AI systems.