Anthropic's AI Agent Claude Struggles to Master Pokémon Red, Highlighting Challenges in AI Development

Why Anthropic's Claude still hasn't beaten Pokémon

In recent months, the AI industry's biggest boosters have started converging on a public expectation that we're on the verge of "artificial general intelligence" (AGI) -- virtual agents that can match or surpass "human-level" understanding and performance on most cognitive tasks. OpenAI is quietly seeding expectations for a "PhD-level" AI agent that could operate autonomously at the level of a "high-income knowledge worker" in the near future. Elon Musk says that "we'll have AI smarter than any one human probably" by the end of 2025. Anthropic CEO Dario Amodei thinks it might take a bit longer but similarly says it's plausible that AI will be "better than humans at almost everything" by the end of 2027. Last month, Anthropic presented its "Claude Plays Pokémon" experiment as a waypoint on the road to that predicted AGI future. It's a project the company said shows "glimmers of AI systems that tackle challenges with increasing competence, not just through training but with generalized reasoning." Anthropic made headlines by trumpeting how Claude 3.7 Sonnet's "improved reasoning capabilities" let the company's latest model make progress in the popular old-school Game Boy RPG in ways "that older models had little hope of achieving." While Claude models from just a year ago struggled even to leave the game's opening area, Claude 3.7 Sonnet was able to make progress by collecting multiple in-game Gym Badges in a relatively small number of in-game actions. That breakthrough, Anthropic wrote, was because the "extended thinking" by Claude 3.7 Sonnet means the new model "plans ahead, remembers its objectives, and adapts when initial strategies fail" in a way that its predecessors didn't. Those things, Anthropic brags, are "critical skills for battling pixelated gym leaders. And, we posit, in solving real-world problems too." But relative success over previous models is not the same as absolute success over the game in its entirety. In the weeks since Claude Plays Pokémon was first made public, thousands of Twitch viewers have watched Claude struggle to make consistent progress in the game. Despite long "thinking" pauses between each move -- during which viewers can read printouts of the system's "reasoning" process -- Claude frequently finds itself pointlessly revisiting completed towns, getting stuck in blind corners of the map for extended periods, or fruitlessly talking to the same unhelpful NPC over and over, to cite just a few examples of distinctly sub-human in-game performance. Watching Claude continue to struggle at a game designed for children, it's hard to imagine we're witnessing the genesis of some sort of computer superintelligence. But even Claude's current sub-human level of Pokémon performance could hold significant lessons for the quest toward generalized, human-level artificial intelligence. Smart in different ways In some sense, it's impressive that Claude can play Pokémon with any facility at all. When developing AI systems that find dominant strategies in games like Go and Dota 2, engineers generally start their algorithms off with deep knowledge of a game's rules and/or basic strategies, as well as a reward function to guide them toward better performance. For Claude Plays Pokémon, though, project developer and Anthropic employee David Hershey says he started with an unmodified, generalized Claude model that wasn't specifically trained or tuned to play Pokémon games in any way. "This is purely the various other things that [Claude] understands about the world being used to point at video games," Hershey told Ars. "So it has a sense of a Pokémon. If you go to claude.ai and ask about Pokémon, it knows what Pokémon is based on what it's read... If you ask, it'll tell you there's eight gym badges, it'll tell you the first one is Brock... it knows the broad structure." In addition to directly monitoring certain key (emulated) Game Boy RAM addresses for game state information, Claude views and interprets the game's visual output much like a human would. But despite recent advances in AI image processing, Hershey said Claude still struggles to interpret the low-resolution, pixelated world of a Game Boy screenshot as well as a human can. "Claude's still not particularly good at understanding what's on the screen at all," he said. "You will see it attempt to walk into walls all the time." Hershey said he suspects Claude's training data probably doesn't contain many overly detailed text descriptions of "stuff that looks like a Game Boy screen." This means that, somewhat surprisingly, if Claude were playing a game with "more realistic imagery, I think Claude would actually be able to see a lot better," Hershey said. "It's one of those funny things about humans that we can squint at these eight-by-eight pixel blobs of people and say, 'That's a girl with blue hair,'" Hershey continued. "People, I think, have that ability to map from our real world to understand and sort of grok that... so I'm honestly kind of surprised that Claude's as good as it is at being able to see there's a person on the screen." Even with a perfect understanding of what it's seeing on-screen, though, Hershey said Claude would still struggle with 2D navigation challenges that would be trivial for a human. "It's pretty easy for me to understand that [an in-game] building is a building and that I can't walk through a building," Hershey said. "And that's [something] that's pretty challenging for Claude to understand... It's funny because it's just kind of smart in different ways, you know?" Where Claude tends to perform better, Hershey said, is in the more text-based portions of the game. During an in-game battle, Claude will readily notice when the game tells it that an attack from an electric-type Pokémon is "not very effective" against a rock-type opponent, for instance. Claude will then squirrel that factoid away in a massive written knowledge base for future reference later in the run. Claude can also integrate multiple pieces of similar knowledge into pretty elegant battle strategies, even extending those strategies into long-term plans for catching and managing teams of multiple creatures for future battles. Claude can even show surprising "intelligence" when Pokémon's in-game text is intentionally misleading or incomplete. "It's pretty funny that they tell you you need to go find Professor Oak next door and then he's not there," Hershey said of an early-game task. "As a five-year-old, that was very confusing to me. But Claude actually typically goes through that same set of motions where it talks to mom, goes to the lab, doesn't find [Oak], says, 'I need to figure something out'... It's sophisticated enough to sort of go through the motions of the way [humans are] actually supposed to learn it, too." These kinds of relative strengths and weaknesses when compared to "human-level" play reflect the overall state of AI research and capabilities in general, Hershey said. "I think it's just a sort of universal thing about these models... We built the text side of it first, and the text side is definitely... more powerful. How these models can reason about images is getting better, but I think it's a decent bit behind." Forget me not Beyond issues parsing text and images, Hershey also acknowledged that Claude can have trouble "remembering" what it has already learned. The current model has a "context window" of 200,000 tokens, limiting the amount of relational information it can store in its "memory" at any one time. When the system's ever-expanding knowledge base fills up this context window, Claude goes through an elaborate summarization process, condensing detailed notes on what it's seen, done, and learned so far into shorter text summaries that lose some of the fine-grained details. This can mean that Claude "has a hard time keeping track of things for a very long time and really having a great sense of what it's tried so far," Hershey said. "You will definitely see it occasionally delete something that it shouldn't have. Anything that's not in your knowledge base or not in your summary is going to be gone, so you have to think about what you want to put there." More than forgetting important history, though, Claude runs into bigger problems when it inadvertently inserts incorrect information into its knowledge base. Like a conspiracy theorist who builds an entire worldview from an inherently flawed premise, Claude can be incredibly slow to recognize when an error in its self-authored knowledge base is leading its Pokémon play astray. "The things that are written down in the past, it sort of trusts pretty blindly," Hershey said. "I have seen it become very convinced that it found the exit to [in-game location] Viridian Forest at some specific coordinates, and then it spends hours and hours exploring a little small square around those coordinates that are wrong instead of doing anything else. It takes a very long time for it to decide that that was a 'fail.'" Still, Hershey said Claude 3.7 Sonnet is much better than earlier models at eventually "questioning its assumptions, trying new strategies, and keeping track over long horizons of various strategies to [see] whether they work or not." While the new model will still "struggle for really long periods of time" retrying the same thing over and over, it will ultimately tend to "get a sense of what's going on and what it's tried before, and it stumbles a lot of times into actual progress from that," Hershey said. "We're getting pretty close..." One of the most interesting things about observing Claude Plays Pokémon across multiple iterations and restarts, Hershey said, is seeing how the system's progress and strategy can vary quite a bit between runs. Sometimes Claude will show it's "capable of actually building a pretty coherent strategy" by "keeping detailed notes about the different paths to try," for instance, he said. But "most of the time it doesn't... most of the time, it wanders into the wall because it's confident it sees the exit." One of the biggest things preventing the current version of Claude from getting better, Hershey said, is that "when it derives that good strategy, I don't think it necessarily has the self awareness to know that one strategy [it] came up with is better than another." And that's not a trivial problem to solve. Still, Hershey said he sees "low-hanging fruit" for improving Claude's Pokémon play by improving the model's understanding of Game Boy screenshots. "I think there's a chance it could beat the game if it had a perfect sense of what's on the screen," Hershey said, saying that such a model would probably perform "a little bit short of human." Expanding the context window for future Claude models will also probably allow those models to "reason over longer time frames and handle things more coherently over a long period of time," Hershey said. Future models will improve by getting "a little bit better at remembering, keeping track of a coherent set of what it needs to try to make progress," he added. Whatever you think about impending improvements in AI models, though, Claude's current performance at Pokémon doesn't make it seem like it's poised to usher in an explosion of human-level, completely generalizable artificial intelligence. And Hershey allows that watching Claude 3.7 Sonnet get stuck on Mt. Moon for 80 hours or so can make it "seem like a model that doesn't know what it's doing." But Hershey is still impressed at the way that Claude's new reasoning model will occasionally show some glimmer of awareness and "kind of tell that it doesn't know what it's doing and know that it needs to be doing something different. And the difference between 'can't do it at all' and 'can kind of do it' is a pretty big one for these AI things for me," he continued. "You know, when something can kind of do something it typically means we're pretty close to getting it to be able to do something really, really well."

Anthropic's AI agent Claude is playing Pokémon and just can't catch 'em all

Anthropic's AI agent Claude is trying to beat Pokémon Red. Apparently, it's no Ash Ketchum. Credit: Warner Bros. Pictures Last month, the $61.5 billion-valuated AI startup Anthropic set up a gaming livestream on Twitch. Gaming livestreams are nothing new on Twitch, but this one is a little different: Claude, Anthropic's AI model, is attempting to beat Pokémon Red. We are now one month in, and the livestream is still going. However, Claude has not progressed all that much. And, at this rate, Anthropic's AI agent may possibly never be the very best, like no one ever was. According to Anthropic, when it first launched the "Claude Plays Pokémon" project, previous versions of its AI agent Claude failed at some very basic tasks. For example, according to Anthropic, Claude 3.5 would try to run away from almost every battle in June 2024. A few months and a few versions of Claude later, Anthropic said there was a stark change. In February 2025, Anthropic gave Claude 3.7 Sonnet a whirl at playing Pokémon. "Within hours, Claude defeated Brock. Days later, it trounced Misty," Anthropic said. "Progress that older models had little hope of achieving." Anthropic said that Claude 3.7 Sonnet could plan ahead, remember objectives, and learn from its mistakes, unlike previous versions of the AI agent. It also built a knowledge base, saw the screen, and simulated button presses. However, the progress Claude 3.7 Sonnet originally made in the game seems to have stalled. For example, livestream viewers watched as Clause 3.7 took 78 hours to get through Mt. Moon in the game. On Reddit, gamers estimated that it would typically take a child just a few hours to advance through the same stage. Claude can be seen going in circles, stumbling around the same paths, and often knocking into walls as it tries to get around the game. The livestream is engaging, especially as a text box lays out Claude's "thinking" as the AI agent tries to figure out what moves to make next. According to Anthropic engineers in an interview with Ars Technica, Claude has an easier time with aspects of the game which involve text, such as Pokémon battles. However, it struggles with the more visual aspects of the game, such as moving around from town to town on the map. Claude 3.7 Sonnet has gone much further in the game than previous Claude models, so there's been progress. However, for those warning that AI will soon be able to take over the world, we're nowhere close to that being a reality yet. Claude still has 151 Pokémon to catch.

One of the World's Most Advanced AI Agents Is Completely Stuck Trying to Beat a Pokémon Game for Children

In case you haven't heard, Anthropic has been livestreaming its AI model, Claude 3.7 Sonnet, attempting to complete a playthrough of Pokémon Red. The experiment, dubbed "Claude Plays Pokémon," is intended to be a demonstration of "AI agents," the industry's ongoing race to create AI models that are capable of operating autonomously by interacting with their environment. Claude has managed to get surprisingly far into the game, clinching three Gym badges and reaching, as of this week, Cerulean City. But it plods along at a painstakingly slow pace, stopping to "think" after every single move, sometimes for longer intervals than others. For nearly 80 agonizing hours, for instance, Claude bumbled cluelessly around Mt. Moon, before finally finding the ladder it needed to escape. Invested Twitch viewers breathed a sigh of relief. Progress isn't looking poised to speed up. The Anthropic AI's excursion through the Kanto region has mostly devolved into running around in circles, unsure of its next move. It needs to hop on Route 5 to reach the next stage, but where and how? A text window in the livestream of Claude's thought process shows that the AI is using a process of elimination to rule out which locations aren't the Route 5 entrance. But will it piece together that it needs to use the HM "Cut" on a few destructible trees to access the fabled path? It's not looking likely: it keeps repeating how it needs to find the "gatehouse" to the route instead. In short, Claude is stuck. One of the AI industry's leading models may well be stumped by a game that's been beaten by literal children for generations. According to engineers, a major challenge for Claude is visually processing what it sees in the game. Claude excels at interpreting the game's text-based portions, including the Pokémon battles. It also has access to the game's RAM to glean information like its in-game coordinates. But it can't consistently interpret the tiny number of pixels that make up its low-res environment. "Claude's still not particularly good at understanding what's on the screen at all," David Hershey, the Anthropic engineer behind the Pokémon experiment, told Ars Technica in a recent interview. "You will see it attempt to walk into walls all the time." Ironically, Hershey suggests, if Claude was playing a more visually realistic game, it might do better. "It's pretty easy for me to understand that [an in-game] building is a building and that I can't walk through a building," Hershey added. "And that's [something] that's pretty challenging for Claude to understand." There are times, however, when Claude is surprisingly clever, like responding to in-game clues that are designed to be misleading. "It's pretty funny that they tell you you need to go find Professor Oak next door and then he's not there," Hershey told Ars, describing one of the first missions in the game. "As a 5-year-old, that was very confusing to me. But Claude actually typically goes through that same set of motions where it talks to mom, goes to the lab, doesn't find [Oak], says, 'I need to figure something out.'" "It's sophisticated enough to sort of go through the motions of the way [humans are] actually supposed to learn it, too," Hershey added. So maybe all is not lost yet. There's still plenty of time for Claude 3.7 Sonnet to turn things around. It's gotten significantly farther than its predecessor 3.0 Sonnet, which couldn't even make it out of Pallet Town, the game's starting area. Still, its struggles show that the technology still has a long way to go to be "agentic," let alone fulfill its promise of one day exceeding human capabilities.