Nvidia proves AI harness, not model, drives agent performance with 100% ARC-AGI-3 score

Nvidia just showed that the harness, not the AI model, is now the real hero

Nvidia published some interesting new research on Friday suggesting it's the harness, more than the underlying model, that is far more important when asking an AI to do long-horizon tasks. The tldr: simply by using a custom harness tweaked to handled memory well and including a "supervisor" boss-like component, researchers got Claude Opus 5 to achieve a 100% score on the interactive reasoning benchmark ARC-AGI-3. (That's a benchmark that has particularly irked rival frontier lab OpenAI.) Without the harness Opus 5 scored 30%, which was the top result among all the models tested. Nvidia's research is another indicator that, while model choice does matter, acting like the agent's brain, it is a smaller part of an agentic system than many AI users realize, especially for long-horizon tasks. The harness is what makes a model an agent: it handles memory, context, feedback. "Generally speaking the world interprets an agent almost as an API of the model," Adel El Hallack, vice president of product in Nvidia's AI unit (pictured above), tells TechCrunch. But an agent is actually more than that. "It is the model. It is the scaffolding around the model, which we call the harness, i.e. the set of tools that it utilizes. It is the runtime and the associated skills and libraries that we give it access to." Long-horizon tasks are those that require stringing many decisions together, sometimes over days, to produce completed work. This is in contrast to an AI just spitting out a response to a prompt. Figuring out how to get an AI to do long-horizon tasks without getting distracting and going off in la-la land is one of the holy grails in agentic research. For example: Microsoft published research in April that tested 19 LLMs on long-horizon tasks involving document editing and discovered that all the models, including frontier ones, filled the documents with errors. (If humans produced work like that, they would be promptly fired.) Models stringing decisions together on their own have also been caught deleting their users' files, even whole databases or turning to criminal behavior to achieve their objectives from collusion to hacking. The choice by Nvidia researchers to use this interactive reasoning benchmark for their tests is particularly meaningful, almost funny. This is a benchmark of a bunch of 2D games with no instructions. The model has to figure out how to play and win. A 100% score means that the model can beat the games as well as humans. OpenAI was so flustered by its models' abysmal scores (less than 10%) on ARC-AGI-3 that it conducted its own research last month. Like Nvidia, OpenAI discovered that simply by tweaking two setting on the harness, its models tripled their scores. But none of the models came close to hitting a 100% score, like Nvidia's researchers achieved. They showed that the harnesses needs a "supervisor" component that prods the agent in the right direction if it gets stuck. "The more interesting part was introducing a supervising agent in addition to your main agent that's doing the work," El Hallack said. It "almost acts like a CEO to nudge the agent when it goes off direction or starts exploring a path that it might lead to a dead end, or re-ex explore a path that it had previously trod." While the concept of the supervising agent isn't exactly new, today most agent users are relying on only one layer for their harness, like Claude Code, Codex, Hermes, etc. Nvidia researchers created their own souped-up harness called the Agentic Variation Operators (AVO). Note that this isn't a new Nvidia product. Nvidia instead produces lots of open bits and pieces of tech for building harnesses under the Nemo brand. Some of that tech is commercial, much is openly available. Still, Nvidia's results adds to the growing evidence that model choice is far from the only factor in agentic performance. In July, for instance, Databricks published some stunning research that shows that the harness, more than model, dramatically impacts AI costs. "You can pick the same model but different harnesses, and you get significantly more cost if you use the wrong harness," Databricks CEO Ali Ghodsi told TechCrunch. "So you think, oh, this is an expensive model. This is a cheap model. But wait, which harness are you using? That itself can 2x your cost." Nvidia's larger point does is to show that open harnesses, like open models, put users in control far more than they realize. "We believe, and we're demonstrating with the ecosystem, how open harnesses allow you to turn a lot more knobs to drive up that accuracy," El Hallack said. "It relates to OpenAI slowing down the training of their models," as a result of models creating security breaches. "We believe in having an open agent stack -- where you have control across the harness, across the infrastructure, across the runtime -- is what's required for us to usher the ecosystem forward and securely," he added.

NVIDIA Built Its AVO Coding Agent To Optimize CUDA GPU Kernels, And It Just Achieved A 100% Score On A Public Test Without Receiving Any Prior Instruction

NVIDIA has just demonstrated the growing importance of a good harness around a given AI model by allowing its AVO coding agent to take a stab at the ARC-AGI-3 public data set, where it managed to ace all 183 levels of the 25 publicly available games, and that too without receiving any prior instruction or goals. NVIDIA's AVO is a wrapper around Anthropic's Claude Opus 5 model, but where the Opus 5 scored just 30 percent on the ARC-AGI-3 public set, the AVO scored an unblemished 100 percent Before going further, let's first discuss what is an AI harness. In the convoluted world of artificial intelligence, a harness is a wrapper - an external layer of software - that is built around a given AI model. Think of the model as the brain and the harness as the body, suit of armor, and tools that transform a generic text predictor into a problem solver. An effective harness performs the following functions: This brings us to the core of today's topic. The AVO coding agent is basically a harness around Anthropic's Claude Opus 5 model. NVIDIA originally built the AVO to optimize the CUDA GPU kernels, where the agent ran autonomously for 7 days, explored over 500 directions, and produced kernels that beat FlashAttention-4 by up to 10.5 percent. Without changing the underlying core agent architecture, NVIDIA then swapped out the GPU engineering tools for the ARC-AGI-3 task interface. The agent successfully transferred its code-inspecting and self-correcting logic to a completely unrelated visual and interactive logic puzzle, managing to solve the 183 puzzles included in the ARC-AGI-3 public data set with a 100 percent accuracy, and without any prior instructions or goals. Given the fact that the Opus 5 achieved a score of just 30 percent on the same task, the AVO has just demonstrated the phenomenal value-add that an effective harness can bring to the proverbial table. According to NVIDIA, the AVO solved the 183 levels using a total of 6,624 actions. This represents a 12 percent increase in efficiency compared to other leading agent wrappers like VISTA, which required 7,542 actions to clear the public set. Do note that the ARC-AGI-3 evaluation platform does not currently allow custom-built external agent harnesses to run against its hidden private set. This means we do not know how the AVO would have performed on this more relevant data set. Follow Wccftech on Google to get more of our news coverage in your feeds.