Salesforce Tackles 'Jagged Intelligence' with New AI Benchmarks for Enterprise Reliability

Salesforce Addresses 'Jagged Intelligence' in AI

Salesforce has launched a series of innovative AI benchmarks and models aimed at tackling the challenge of 'jagged intelligence' in artificial intelligence systems. This phenomenon refers to the discrepancy between an AI model's raw intelligence and its ability to perform consistently in real-world, unpredictable enterprise environments 1

1

2

New Benchmarks for AI Reliability

To address this issue, Salesforce has introduced several new benchmarks:

1. SIMPLE Benchmark: A public dataset featuring 225 straightforward reasoning questions that are easy for humans but challenging for AI. This benchmark aims to quantify the 'jaggedness' of AI models and improve their real-world performance 1

   1

   .

2. ContextualJudgeBench: This benchmark evaluates AI-enabled judges rather than the models themselves, focusing on the reliability of AI systems that assess other models 1

   1

   .

3. CRMArena: A framework designed to evaluate how AI agents perform in customer relationship management (CRM) tasks, such as summarizing sales emails and making commerce recommendations 1

   1

   2

   2

   .

Enterprise General Intelligence (EGI)

Salesforce is pushing towards what they call "Enterprise General Intelligence" (EGI), which focuses on developing AI specifically for business complexity. This approach aims to create purpose-built AI agents optimized for both capability and consistency in business environments 2

2

Related Stories

Salesforce Unveils Agentforce 2.0: A Leap Forward in Enterprise AI and Digital Labor

Salesforce AI gains 6,000 customers while quietly pulling back on large language models

AI Agents in Enterprise: Salesforce and Outreach Unveil New Platforms with Surprising Insights

Salesforce Launches Agentforce Testing Center: A New Era in AI Agent Lifecycle Management

New AI Models and Embeddings

Salesforce has also introduced new AI models and embeddings to enhance enterprise AI capabilities:

1. SFR-Embedding: A new model for deeper contextual understanding, leading the Massive Text Embedding Benchmark (MTEB) across 56 datasets 2

   2

   .

2. SFR-Embedding-Code: A specialized version for developers, enabling high-quality code search and streamlining development 2

   2

   .

3. xLAM V2 (Large Action Model): A family of models designed to predict actions rather than just generate text, starting at just 1 billion parameters 2

   2

   .

Implications for Business AI Applications

These developments have significant implications for businesses looking to implement AI:

1. Improved Consistency: By addressing 'jagged intelligence', Salesforce aims to create AI systems that perform more reliably in unpredictable business environments 1

   1

   2

   2

   .

2. Enhanced Trust: Better benchmarks and more consistent performance could lead to higher trust from business leaders in implementing AI systems 1

   1

   .

3. Tailored Solutions: The focus on EGI and CRM-specific benchmarks suggests a move towards AI solutions tailored for specific business needs 1

   1

   2

   2

   .

4. Efficient Models: Smaller, action-focused AI models like xLAM V2 may outperform larger language models for specific business tasks, offering more efficient solutions 2

   2

   .

As AI continues to evolve, Salesforce's research lays the groundwork for more reliable, efficient, and business-focused AI agents. This could potentially revolutionize how enterprises leverage AI technology in their operations, leading to significant productivity gains and improved decision-making processes.