ChatGPT Loses Chess Match to 1970s Atari 2600, Raising Questions About AI Limitations

How Did ChatGPT Get 'Absolutely Wrecked' at Chess by an 1970s-Era Atari 2600?

OpenAI's ChatGPT has some major AI chatbot competitors in the market: Gemini, Copilot, Claude. Now add to that list the Atari 2600. The OG video game console, which was first released in 1977, was used in an engineer's experiment to see how it would fare playing chess against the AI chatbot. By using a software emulator to run Atari's 1979 game Video Chess, Citrix engineer Robert Caruso said he was able to set up a match between ChatGPT and the 46-year-old game. The matchup did not go well for ChatGPT. "ChatGPT confused rooks for bishops, missed pawn forks and repeatedly lost track of where pieces were -- first blaming the Atari icons as too abstract, then faring no better even after switching to standard chess notations," Caruso wrote in a LinkedIn post. "It made enough blunders to get laughed out of a 3rd-grade chess club," Caruso said. "ChatGPT got absolutely wrecked at the beginner level." Caruso wrote that the 90-minute match continued badly and that the AI chatbot repeatedly requested that the match start over. For decades, the ability for computers to defeat humans at chess has been a measure of their power. In 1997, IBM made headlines when its Deep Blue technology defeated chess grandmaster Garry Kasparov in a series of matches. Caruso's experiment doesn't mean ChatGPT is useless for chess, but because it's more of a language model than a supercomputer, it's less likely to serve that purpose well. A few years ago, a developer created a ChatGPT plugin called ChessGPT. But it may be better to discuss chess with OpenAI's chatbot than to try to play against it. A representative for OpenAI did not immediately return a request for comment. (Disclosure: Ziff Davis, CNET's parent company, in April filed a lawsuit against OpenAI, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.)

ChatGPT asked to play an Atari 2600 at chess then 'got absolutely wrecked on the beginner level'

An engineer toying around with ChatGPT found OpenAI's apparently world-leading LLM getting a little bolshy about how it would do at chess. In fact, ChatGPT itself asked Citrix engineer Robert Caruso to set it up against a basic chess program to see "how quickly" it would win: and then proceeded to get battered by an Atari 2600. First things first: chess engines are now unquestionably superior to human players, and an off-the-shelf program like Stockfish will handily trounce the best in the world. There are also AI-based chess engines from the likes of DeepMind. And ChatGPT 4o, the latest model, may be a leader in LLMs -- but it is not a chess engine. Nevertheless, you might expect something a little more impressive than this. Talking to ChatGPT about the history of AI in chess "led to it volunteering to play Atari Chess," said Caruso on LinkedIn. "It wanted to find out how quickly it could beat a game that only thinks 1-2 moves ahead on a 1.19 MHz CPU." And? "ChatGPT got absolutely wrecked on the beginner level," says Caruso. "Despite being given a baseline board layout to identify pieces, ChatGPT confused rooks for bishops, missed pawn forks, and repeatedly lost track of where pieces were -- first blaming the Atari icons as too abstract to recognize, then faring no better even after switching to standard chess notation. It made enough blunders to get laughed out of a 3rd grade chess club." Video Chess is as basic as chess software comes, which is entirely a function of its era: the major challenge for the programmers was creating a working engine within 4KB (which was still double the standard 2KB for other VCS games). It essentially brute forces the best move in a given position, but lacks an overall strategy and doesn't think ahead. A decent human player, in other words, should have a pretty easy time conquering Video Chess. But for 90 minutes Caruso "had to stop [ChatGPT] from making awful moves and correct its board awareness multiple times per turn. It kept promising it would improve 'if we just started over.' Eventually, even ChatGPT knew it was beat -- and conceded with its head hung low." ChatGPT itself asked for the game of chess against an Atari, "which it proclaimed it would easily win," after a conversation about Stockfish and AlphaZero. The LLM was apparently "curious how quickly it could win" and, because Caruso had told it he was a weak player, "offered to teach me strategy along the way." The story isn't entirely one-sided. Caruso says that when ChatGPT had an accurate sense of the board it offered him some "solid guidance" and at times was "genuinely impressive." But at others, and this will be familiar to anyone who's spent much time fooling around with ChatGPT, "it made absurd suggestions... or tried to move pieces that had already been captured, even during turns when it otherwise had an accurate view of the board." Naturally the AI evangelists will be out in force to say this is meaningless, it's not what LLMs are designed to do, and so on. But this does raise wider questions about the technology and particularly its understanding of context (or lack thereof). "Its inability to retain a basic board state from turn to turn was very disappointing," says Caruso. "Is that really any different from forgetting other crucial context in a conversation?" In a nod to Atari's once-famous marketing slogan, Caruso signs off: "Have you played Atari today? ChatGPT wishes it hadn't."