Patronus AI unveils Generative Simulators to fix 63% failure rate plaguing AI agents

AI agents fail 63% of the time on complex tasks. Patronus AI says its new 'living' training worlds can fix that.

Patronus AI, the artificial intelligence evaluation startup backed by $20 million from investors including Lightspeed Venture Partners and Datadog, unveiled a new training architecture Tuesday that it says represents a fundamental shift in how AI agents learn to perform complex tasks. The technology, which the company calls "Generative Simulators," creates adaptive simulation environments that continuously generate new challenges, update rules dynamically, and evaluate an agent's performance as it learns -- all in real time. The approach marks a departure from the static benchmarks that have long served as the industry standard for measuring AI capabilities but have increasingly come under fire for failing to predict real-world performance. "Traditional benchmarks measure isolated capabilities, but they miss the interruptions, context switches, and layered decision-making that define real work," said Anand Kannappan, chief executive and co-founder of Patronus AI, in an exclusive interview with VentureBeat. "For agents to perform at human levels, they need to learn the way humans do -- through dynamic experience and continuous feedback." The announcement arrives at a critical moment for the AI industry. AI agents are reshaping software development, from writing code to carrying out complex instructions. Yet LLM-based agents are prone to errors and often perform poorly on complicated, multi-step tasks. Research published earlier this year found that an agent with just a 1% error rate per step can compound to a 63% chance of failure by the hundredth step -- a sobering statistic for enterprises seeking to deploy autonomous AI systems at scale. Why static AI benchmarks are failing -- and what comes next Patronus AI's approach addresses what the company describes as a growing mismatch between how AI systems are evaluated and how they actually perform in production. Traditional benchmarks, the company argues, function like standardized tests: they measure specific capabilities at a fixed point in time but struggle to capture the messy, unpredictable nature of real work. The new Generative Simulators architecture flips this model. Rather than presenting agents with a fixed set of questions, the system generates assignments, environmental conditions, and oversight processes on the fly, then adapts based on how the agent behaves. "Over the past year, we've seen a shift away from traditional static benchmarks toward more interactive learning grounds," Rebecca Qian, chief technology officer and co-founder of Patronus AI, told VentureBeat. "This is partly because of the innovation we've seen from model developers -- the shift toward reinforcement learning, post-training, and continual learning, and away from supervised instruction tuning. What that means is there's been a collapse in the distinction between training and evaluation. Benchmarks have become environments." The technology builds on reinforcement learning -- an approach where AI systems learn through trial and error, receiving rewards for correct actions and penalties for mistakes. Reinforcement learning is an approach where AI systems learn to make optimal decisions by receiving rewards or penalties for their actions, improving through trial and error. RL can help agents improve, but it typically requires developers to extensively rewrite their code. This discourages adoption, even though the data these agents generate could significantly boost performance through RL training. Patronus AI also introduced a new concept it calls "Open Recursive Self-Improvement," or ORSI -- environments where agents can continuously improve through interaction and feedback without requiring a complete retraining cycle between attempts. The company positions this as critical infrastructure for developing AI systems capable of learning continuously rather than being frozen at a point in time. Inside the 'Goldilocks Zone': How adaptive AI training finds the sweet spot At the heart of Generative Simulators lies what Patronus AI calls a "curriculum adjuster" -- a component that analyzes agent behavior and dynamically modifies the difficulty and nature of training scenarios. The approach draws inspiration from how effective human teachers adapt their instruction based on student performance. Qian explained the approach using an analogy: "You can think of this as a teacher-student model, where we're training the model and the professor continually adapts the curriculum." This adaptive approach addresses a problem that Kannappan described as finding the "Goldilocks Zone" in training data -- ensuring that examples are neither too easy nor too hard for a given model to learn from effectively. "What's important is not just whether you can train on a data set, but whether you can train on a high-quality data set that's tuned to your model -- one it can actually learn from," Kannappan said. "We want to make sure the examples aren't too hard for the model, nor too easy." The company says initial results show meaningful improvements in agent performance. Training on Patronus AI's environments has increased task completion rates by 10% to 20% across real-world tasks including software engineering, customer service, and financial analysis, according to the company. The AI cheating problem: How 'moving target' environments prevent reward hacking One of the most persistent challenges in training AI agents through reinforcement learning is a phenomenon researchers call "reward hacking" -- where systems learn to exploit loopholes in their training environment rather than genuinely solving problems. Famous examples include early agents that learned to hide in corners of video games rather than actually play them. Generative Simulators addresses this by making the training environment itself a moving target. "Reward hacking is fundamentally a problem when systems are static. It's like students learning to cheat on a test," Qian said. "But when we're continually evolving the environment, we can actually look at parts of the system that need to adapt and evolve. Static benchmarks are fixed targets; generative simulator environments are moving targets." Patronus AI reports 15x revenue growth as enterprise demand for agent training surges Patronus AI positions Generative Simulators as the foundation for a new product line it calls "RL Environments" -- training grounds designed for foundation model laboratories and enterprises building agents for specific domains. The company says this offering represents a strategic expansion beyond its original focus on evaluation tools. "We've grown 15x in revenue this year, largely due to the high-quality environments we've developed that have been shown to be extremely learnable by different kinds of frontier models," Kannappan said. The CEO declined to specify absolute revenue figures but said the new product has allowed the company to "move higher up the stack in terms of where we sell and who we sell to." The company's platform is used by numerous Fortune 500 enterprises and leading AI companies around the world. Why OpenAI, Anthropic, and Google can't build everything in-house A central question facing Patronus AI is why the deep-pocketed laboratories developing frontier models -- organizations like OpenAI, Anthropic, and Google DeepMind -- would license training infrastructure rather than build it themselves. Kannappan acknowledged that these companies "are investing significantly in environments" but argued that the breadth of domains requiring specialized training creates a natural opening for third-party providers. "They want to improve agents on lots of different domains, whether it's coding or tool use or navigating browsers or workflows across finance, healthcare, energy, and education," he said. "Solving all those different operational problems is very difficult for a single company to do." The competitive landscape is intensifying. Microsoft recently released Agent Lightning, an open-source framework that makes reinforcement learning work for any AI agent without rewrites. NVIDIA's NeMo Gym offers modular RL infrastructure for developing agentic AI systems. Meta researchers released DreamGym in November, a framework that simulates RL environments and dynamically adjusts task difficulty as agents improve. 'Environments are the new oil': Patronus AI's audacious bet on the future of AI training Looking ahead, Patronus AI frames its mission in sweeping terms. The company wants to "environmentalize all of the world's data" -- converting human workflows into structured systems that AI can learn from. "We think that everything should be an environment -- internally, we joke that environments are the new oil," Kannappan said. "Reinforcement learning is just one training method, but the construct of an environment is what really matters." Qian described the opportunity in expansive terms: "This is an entirely new field of research, which doesn't happen every day. Generative simulation is inspired by early research in robotics and embodied agents. It's been a pipe dream for decades, and we're only now able to achieve these ideas because of the capabilities of today's models." The company launched in September 2023 with a focus on evaluation -- helping enterprises identify hallucinations and safety issues in AI outputs. That mission has now expanded upstream into training itself. Patronus AI argues that the traditional separation between evaluation and training is collapsing -- and that whoever controls the environments where AI agents learn will shape their capabilities. "We are really at this critical point, this inflection point, where what we do right now will impact what the world is going to look like for generations to come," Qian said. Whether Generative Simulators can deliver on that promise remains to be seen. The company's 15x revenue growth suggests enterprise customers are hungry for solutions, but deep-pocketed players from Microsoft to Meta are racing to solve the same fundamental problem. If the last two years have taught the industry anything, it's that in AI, the future has a habit of arriving ahead of schedule.

Patronus AI's debuts Generative Simulators to support continuous evolution and improvement of AI agents - SiliconANGLE

Patronus AI's debuts Generative Simulators to support continuous evolution and improvement of AI agents Artificial intelligence model training and testing tools startup Patronus AI Inc. today announced the availability of a new offering called "Generative Simulators" that are designed to help evaluate and improve autonomous AI agents. The new simulators are a core element of Patronus AI's reinforcement learning environments, which are simulated worlds that enable thorough testing of AI agents. They can adapt these simulations on the fly, continuously creating new tasks, scenarios and rules to ensure that AI agents are constantly learning new things and never go stale. Within Patronus AI's RL environments, AI agents can learn new skills and capabilities through trial-and-error within a virtual setting that mimics real-world workflows. Each environment incorporates domain-specific rules, best practices and verifiable rewards that incentivize AI agents to optimize their performance on a range of work-related tasks. They enable developers to expose agents to new kinds of reasoning challenges and interruptions, so they can evolve over time. They also serve to evaluate the skills of AI agents. The startup says training AI agents and evolving them over time remains a key challenge for foundation model labs. AI agents are designed to perform tasks autonomously with minimal human supervision, and are therefore a whole different ball game compared to standard generative AI chatbots. One of the main problems is that the static tests and training data used to create the large language models that power AI agents do not reflect the dynamic and interactive nature of real-world workflows. As a result, agents that perform well on static benchmarks can fall apart when they're deployed in the real world and the requirements of a task evolve. Agents must also learn to use third-party tools successfully and stay on track over long periods of time. Patronus AI co-founder and Chief Executive Anand Kannappan said traditional benchmarks are good for measuring isolated capabilities of AI models, but they don't take into account the constant context switching, interruptions and multi-layered decision-making that occurs when they're doing work. "For agents to perform tasks at human-comparable levels, they need to learn the way humans do - through dynamic, feedback-driven experience that captures real-world nuance," he said. Evaluations are performed by Patronus AI's Glider LLM, which was purpose-built as a fast, impartial and highly flexible "judge" for third-party AI models. If any improvements are required, these can be carried out by Percival, a second model developed by the company that's designed to find and automatically fix AI malfunctions. Percival automates this process, analyzing agent's workflows to identify any specific substeps within them that cause problems before suggesting a way to fix it. The new Generative Simulators are meant to facilitate this kind of learning. They can generate new "assignments" for agents alongside the surrounding conditions, oversight process and so on, and then adapt these continuously based on the agent's behavior. So instead of a fixed training environment, they act more like a "living practice world" that continually creates newer and more relevant challenges and feedback. As a result, AI agents never stop learning and improving, the company said. The simulators also support a new training technique Patronus AI has devised that's called Open Recursive Self-Improvement or ORSI. Within its training environments, ORSI allows agents to improve their performance on new tasks through interactions and feedback, without the need for a full retraining cycle between attempts. "When a coding agent can decompose a complex task, handle distractions mid-implementation, coordinate with teammates on priorities and verify its work, that's when we're seeing true value," said Patronus AI Chief Technology Officer Rebecca Qian. "Our RL environments give foundation model labs the training infrastructure to develop agents that don't just perform well on predefined tests, but work in the real world."