AI Models Struggle with Abstract Visual Reasoning, Falling Short of Human Capabilities

AI Models Face Challenges in Abstract Visual Reasoning

Researchers from the USC Viterbi School of Engineering Information Sciences Institute (ISI) have conducted a groundbreaking study to assess the capabilities of artificial intelligence in solving abstract visual puzzles similar to those found in human IQ tests. The study, presented at the Conference on Language Modeling (COLM 2024) in Philadelphia, reveals significant limitations in AI's ability to perform nonverbal abstract reasoning tasks 1

1

Study Methodology and Findings

The research team, led by Kian Ahrabian and Zhivar Sourati, tested 24 different multi-modal large language models (MLLMs) using puzzles based on Raven's Progressive Matrices, a standard test of abstract reasoning. The results showed a stark contrast between open-source and closed-source AI models 2

2

Open-source models performed poorly, with Ahrabian stating, "They were really bad. They couldn't get anything out of it." In contrast, closed-source models like GPT-4V demonstrated better performance, though still far from matching human cognitive abilities 3

3

Identifying AI's Stumbling Blocks

The researchers delved deeper to understand where the AI models were failing. They discovered that the issue was not limited to visual processing but extended to the reasoning process itself. Even when provided with detailed textual descriptions of the images, many models struggled to reason effectively 4

4

Related Stories

New AGI Benchmark Stumps Leading AI Models, Highlighting Gap in General Intelligence

Apple Study Reveals Limitations in AI's Mathematical Reasoning Abilities

Study Reveals GPT Models Struggle with Flexible Reasoning, Highlighting Limitations in AI Cognition

Apple Study Challenges AI Reasoning Capabilities, Casting Doubt on AGI Claims

Improving AI Performance

To enhance AI performance, the team explored a technique called "Chain of Thought prompting." This method guides the AI through step-by-step reasoning tasks and led to significant improvements in some cases. Ahrabian noted, "By guiding the models with hints, we were able to see up to 100% improvement in performance" 2

2

Implications and Future Directions

Jay Pujara, research associate professor and author of the study, emphasized the importance of understanding AI's limitations: "We still have such a limited understanding of what new AI models can do, and until we understand these limitations, we can't make AI better, safer, and more useful" 1

1

The study's findings highlight both the current limitations of AI and the potential for future advancements. As AI models continue to evolve, this research could pave the way for developing AI systems that can not only understand but also reason in ways more comparable to human cognition 4

4