MIT Study Reveals Nuances in Human-AI Collaboration Effectiveness

Do humans work better with AI? Study shows which tasks benefit most

With the rise of AI, a new study from MIT researchers looked into which tasks could benefit most from human-AI collaboration and which are best handled independently. As artificial intelligence (AI) advances, there has been a global push for humans to learn to work alongside it to keep pace with its rapid adoption in both personal and professional settings. However, a recent study from MIT's Center for Collective Intelligence (CCI) found that while human-AI collaboration can be useful, certain tasks may yield better results when handled independently. "There's a prevailing assumption that integrating AI into a process will always help performance - but we show that that isn't true," Michelle Vaccaro, lead author of the study and a doctoral student at MIT, said in a statement. "In some cases, it's beneficial to leave some tasks solely for humans, and some tasks solely for AI," she added. To gain a clearer understanding of when humans and AI work most effectively together, the researchers analysed 370 results drawn from more than 100 experimental studies involving human-AI combinations across various tasks. The studies covered three approaches to handling a task, which included humans alone, AI systems alone, and human-AI teams. The researchers found that while human-AI teams tended to outperform humans working independently, they did not exceed the results achieved by AI systems working alone. Published in the journal Nature Human Behaviour, the study also found no evidence of "human-AI synergy," meaning that for certain tasks, relying solely on either humans or AI could produce better outcomes than collaboration. But the meta-analysis found some specific cases where human-AI collaboration could be more effective than humans or AI alone. "Even though our main result suggests that - on average - combining humans and AI leads to performance losses, we do not think this means that combining humans and AI is a bad idea," the authors wrote in the paper. "On the contrary, we think it just means that future work needs to focus more specifically on finding effective processes that integrate humans and AI. Our other results suggest promising ways to proceed". The researchers identified decision-making tasks, such as detecting deepfakes and diagnosing medical cases, where human-AI teams underperformed compared to AI working independently. In contrast, they also found that humans and AI worked better together on creative tasks, like generating new visual or written content. In these cases, the human-AI teams even surpassed the performance of the best-performing individual humans or AI systems. The researchers theorised that this could be due to the nature of creative work itself. Creative tasks, they explained, require a mix of creativity, knowledge, and insight, which are qualities that are inherently human and difficult for AI to fully replicate. Yet those tasks also require repetitive, routine processes where AI is highly effective. As a result, a combination of human and AI strengths could have allowed for better outcomes in creative work. "Let AI handle the background research, pattern recognition, predictions, and data analysis, while harnessing human skills to spot nuances and apply contextual understanding," Thomas Malone, co-author of the study and director of the MIT CCI, said in a statement. He concluded: "As we continue to explore the potential of these collaborations, it's clear that the future lies not just in replacing humans with AI, but also in finding innovative ways for them to work together effectively".

Humans and AI: Do they work better together or alone?

The potential of human-AI collaboration has captured our imagination: a future where human creativity and AI's analytical power combine to make critical decisions and solve complex problems. But new research from the MIT Center for Collective Intelligence (CCI) suggests this vision may be much more nuanced than we once thought. Published in Nature Human Behaviour, "When Combinations of Humans and AI Are Useful" is the first large-scale meta-analysis conducted to better understand when human-AI combinations are useful in task completion, and when they are not. Surprisingly, the research has found that combining humans and AI to complete decision-making tasks often fell short; but human-AI teams showed much potential working in combination to perform creative tasks. The research, conducted by MIT doctoral student and CCI affiliate Michelle Vaccaro, and MIT Sloan School of Management professors Abdullah Almaatouq and Thomas Malone, arrives at a time marked by both excitement and uncertainty about AI's impact on the workforce. Instead of focusing on job displacement predictions, Malone said that he and the team wanted to explore questions they believe deserve more attention: When do humans and AI work together most effectively? And how can organizations create guidelines and guardrails to ensure these partnerships succeed? The researchers conducted a meta-analysis of 370 results on AI and human combinations in a variety of tasks from 106 different experiments published in relevant academic journals and conference proceedings between January 2020 and June 2023. All the studies compared three different ways of performing tasks: a) human-only systems b) AI-only systems, and c) human-AI collaborations. The overall goal of the meta-analysis was to understand the underlying trends revealed by the combination of the studies. Test outcomes The researchers found that on average, human-AI teams performed better than humans working alone, but didn't surpass the capabilities of AI systems operating on their own. Importantly, they did not find "human-AI synergy," which means that the average human-AI systems performed worse than the best of humans alone or AI alone on the performance metrics studied. This suggests that using either humans alone or AI systems alone would have been more effective than the human-AI collaborations studied. "There's a prevailing assumption that integrating AI into a process will always help performance -- but we show that that isn't true," said Vaccaro. "In some cases, it's beneficial to leave some tasks solely for humans, and some tasks solely for AI." The team also identified factors affecting how well humans and AI work together. For instance, for decision-making tasks like classifying deep fakes, forecasting demand, and diagnosing medical cases, human-AI teams often underperformed against AI alone. However, for many creative tasks, such as summarizing social media posts, answering questions in a chat, or generating new content and imagery, these collaborations were often better than the best of humans or AI working independently. "Even though AI in recent years has mostly been used to support decision-making by analyzing large amounts of data, some of the most promising opportunities for human-AI combinations now are in supporting the creation of new content, such as text, images, music, and video," said Malone. The team theorized that this advantage in creative endeavors stems from their dual nature: While these tasks require human talents like creativity, knowledge, and insight, they also involve repetitive work where AI excels. Designing an image, for instance, requires both artistic inspiration -- where humans excel -- and detailed execution -- where AI often shines. In a similar vein, writing and generating many kinds of text documents requires human knowledge and insight, but also involves routine and automated processes such as filling in boilerplate text. "There is a lot of potential in combining humans and AI, but we need to think more critically about it," said Vaccaro. "The effectiveness is not necessarily about the baseline performance of either of them, but about how they work together and complement each other." Optimizing collaboration The research team believes its findings provide guidance and lessons for organizations looking to bring AI into their workplaces more effectively. For starters, Vaccaro emphasized the importance of assessing whether humans and AI are truly outperforming either humans or AI working independently. "Many organizations may be overestimating the effectiveness of their current systems," she added. "They need to get a pulse on how well they're working." Next, they need to evaluate where AI can help workers. The study indicates that AI can be particularly helpful in creative tasks, so organizations should explore what kinds of creative work could be ripe for the insertion of AI. Finally, organizations need to set clear guidelines and establish robust guardrails for AI usage. They might, for example, devise processes that leverage complementary strengths. "Let AI handle the background research, pattern recognition, predictions, and data analysis, while harnessing human skills to spot nuances and apply contextual understanding," Malone suggested. In other words, "Let humans do what they do best." Malone concluded, "As we continue to explore the potential of these collaborations, it's clear that the future lies not just in replacing humans with AI, but also in finding innovative ways for them to work together effectively."

Do humans and AI work better together? - Earth.com

Have you ever wondered how the future would be with humans and AI working together to solve complex problems and make critical decisions? This intriguing subject has been the focus for researchers at the MIT Center for Collective Intelligence (CCI). Their latest findings might have you looking at this idealistic vision with a more discerning eye. The team included doctoral student Michelle Vaccaro, Professor Abdullah Almaatouq, and Professor Thomas Malone. The researchers chose to steer clear of the often-debated topic of AI's potential to replace humans in certain jobs. Instead, they wanted to delve into questions such as: When do humans and AI work together most effectively? How can organizations ensure these partnerships succeed? The study authors noted that people are increasingly working with artificial intelligence (AI) tools in fields such as medicine, finance and law, as well as in daily activities such as traveling, shopping, and communicating. "These human-AI systems have tremendous potential given the complementary nature of humans and AI - the general intelligence of humans allows us to reason about diverse problems, and the computational power of AI systems allows them to accomplish specific tasks that people find difficult." The researchers conducted a meta-analysis that spanned over 370 results from 106 different experiments. The experiments compared three different ways of performing tasks - with only humans, with only AI, and with collaborations. What the researchers discovered was a fascinating contradiction to the prevailing belief that the symbiosis of humans and AI would always lead to better outcomes. Human-AI teams indeed performed better than humans alone. However, they did not surpass the capabilities of AI systems operating on their own. "There's a prevailing assumption that integrating AI into a process will always help performance - but we show that that isn't true," said Vaccaro. "In some cases, it's beneficial to leave some tasks solely for humans, and some tasks solely for AI." The type of task also played a significant role in the effectiveness of collaboration. Decision-making tasks like classifying deep fakes, forecasting demand, and diagnosing medical cases were better tackled by AI alone. Yet, when looking at creative tasks like summarizing social media posts or generating new content and imagery, it was found that human-AI teams often outperformed both people and AI working independently. "Even though AI in recent years has mostly been used to support decision-making by analyzing large amounts of data, some of the most promising opportunities for human-AI combinations now are in supporting the creation of new content, such as text, images, music, and video," said Malone. The researchers attributed this unique advantage in creative tasks to their dual nature. While creativity, knowledge, and insight are vital human talents in these tasks, they also involve repetitive work where AI often excels. "There is a lot of potential in combining humans and AI, but we need to think more critically about it," said Vaccaro. "The effectiveness is not necessarily about the baseline performance of either of them, but about how they work together and complement each other." This study's findings offer crucial insights for organizations aiming at successfully integrating AI into their workplaces. The research team advises organizations to assess whether collaborations would actually outperform either entity working independently. ""Many organizations may be overestimating the effectiveness of their current systems," noted Vaccaro. "They need to get a pulse on how well they're working." Additionally, organizations should evaluate how AI could assist their workers, especially in creative tasks, and establish robust guidelines for AI usage. The idea should be to leverage the complementary strengths. "Let AI handle the background research, pattern recognition, predictions, and data analysis, while harnessing human skills to spot nuances and apply contextual understanding," Malone suggested. In other words: "Let humans do what they do best." As the researchers continue to explore the potential of artifical intelligence, they firmly believe that the future does not lie in replacing humans, but in finding innovative ways for both to work together effectively. Like what you read? Subscribe to our newsletter for engaging articles, exclusive content, and the latest updates.

Humans and AI: Do they work better together or alone?

Cambridge, Mass., Oct. 28, 2024 (GLOBE NEWSWIRE) -- The potential of human-AI collaboration has captured our imagination: a future where human creativity and AI's analytical power combine to make critical decisions and solve complex problems. But new research from the MIT Center for Collective Intelligence (CCI) suggests this vision may be much more nuanced than we once thought. Published today in Nature Human Behaviour, "When Combinations of Humans and AI Are Useful" is the first large-scale meta-analysis conducted to better understand when human-AI combinations are useful in task completion, and when they are not. Surprisingly, the research has found that combining humans and AI to complete decision-making tasks often fell short; but human-AI teams showed much potential working in combination to perform creative tasks. The research, conducted by MIT doctoral student and CCI affiliate Michelle Vaccaro, and MIT Sloan School of Management professors Abdullah Almaatouq and Thomas Malone, arrives at a time marked by both excitement and uncertainty about AI's impact on the workforce. Instead of focusing on job displacement predictions, Malone said that he and the team wanted to explore questions they believe deserve more attention: When do humans and AI work together most effectively? And how can organizations create guidelines and guardrails to ensure these partnerships succeed? The researchers conducted a meta-analysis of 370 results on AI and human combinations in a variety of tasks from 106 different experiments published in relevant academic journals and conference proceedings between January 2020 and June 2023. All the studies compared three different ways of performing tasks: a.) human-only systems b.) AI-only systems, and c.) human-AI collaborations. The overall goal of the meta-analysis was to understand the underlying trends revealed by the combination of the studies. Test Outcomes The researchers found that on average, human-AI teams performed better than humans working alone, but didn't surpass the capabilities of AI systems operating on their own. Importantly, they did not find "human-AI synergy," which means that the average human-AI systems performed worse than the best of humans alone or AI alone on the performance metrics studied. This suggests that using either humans alone or AI systems alone would have been more effective than the human-AI collaborations studied. "There's a prevailing assumption that integrating AI into a process will always help performance -- but we show that that isn't true," said Vaccaro. "In some cases, it's beneficial to leave some tasks solely for humans, and some tasks solely for AI." The team also identified factors affecting how well humans and AI work together. For instance, for decision-making tasks like classifying deep fakes, forecasting demand, and diagnosing medical cases, human-AI teams often underperformed against AI alone. However, for many creative tasks, such as summarizing social media posts, answering questions in a chat, or generating new content and imagery, these collaborations were often better than the best of humans or AI working independently. "Even though AI in recent years has mostly been used to support decision-making by analyzing large amounts of data, some of the most promising opportunities for human-AI combinations now are in supporting the creation of new content, such as text, images, music, and video," said Malone. The team theorized that this advantage in creative endeavors stems from their dual nature: While these tasks require human talents like creativity, knowledge, and insight, they also involve repetitive work where AI excels. Designing an image, for instance, requires both artistic inspiration -- where humans excel -- and detailed execution -- where AI often shines. In a similar vein, writing and generating many kinds of text documents requires human knowledge and insight, but also involves routine, and automated processes such as filling in boilerplate text. "There is a lot of potential in combining humans and AI, but we need to think more critically about it," said Vaccaro. "The effectiveness is not necessarily about the baseline performance of either of them, but about how they work together and complement each other." Optimizing collaboration The research team believes its findings provide guidance and lessons for organizations looking to bring AI into their workplaces more effectively. For starters, Vaccaro emphasized the importance of assessing whether humans and AI are truly outperforming either humans or AI working independently. "Many organizations may be overestimating the effectiveness of their current systems," she added. "They need to get a pulse on how well they're working. Next, they need to evaluate where AI can help workers. The study indicates that AI can be particularly helpful in creative tasks, so organizations should explore what kinds of creative work could be ripe for the insertion of AI. Finally, organizations need to set clear guidelines and establish robust guardrails for AI usage. They might, for example, devise processes that leverage complementary strengths. "Let AI handle the background research, pattern recognition, predictions, and data analysis, while harnessing human skills to spot nuances and apply contextual understanding," Malone suggested. In other words: "Let humans do what they do best." Malone concluded: "As we continue to explore the potential of these collaborations, it's clear that the future lies not just in replacing humans with AI, but also in finding innovative ways for them to work together effectively." Attachment Humans and AI Casey Bayer MIT Sloan School of Management 914.584.9095 bayerc@mit.edu Patricia Favreau MIT Sloan School of Management 617.595.8533 pfavreau@mit.edu Market News and Data brought to you by Benzinga APIs

When Human-AI Teams Thrive and When They Don't - Neuroscience News

Summary: A new study reveals that while human-AI collaboration can be powerful, it depends on the task. Analysis of hundreds of studies found that AI outperformed human-AI teams in decision-making tasks, while collaborative teams excelled in creative tasks like content generation. This research suggests organizations may overestimate the benefits of human-AI synergy. Instead, strategic use of AI's strengths in data processing and humans' creativity may yield the best results. These findings can help shape AI guidelines that enhance performance by maximizing complementary skills. Researchers argue that the future of work lies in nuanced collaboration rather than across-the-board AI adoption. The potential of human-AI collaboration has captured our imagination: a future where human creativity and AI's analytical power combine to make critical decisions and solve complex problems. But new research from the MIT Center for Collective Intelligence (CCI) suggests this vision may be much more nuanced than we once thought. Published today in Nature Human Behaviour, "When Combinations of Humans and AI Are Useful" is the first large-scale meta-analysis conducted to better understand when human-AI combinations are useful in task completion, and when they are not. Surprisingly, the research has found that combining humans and AI to complete decision-making tasks often fell short; but human-AI teams showed much potential working in combination to perform creative tasks. The research, conducted by MIT doctoral student and CCI affiliate Michelle Vaccaro, and MIT Sloan School of Management professors Abdullah Almaatouq and Thomas Malone, arrives at a time marked by both excitement and uncertainty about AI's impact on the workforce. Instead of focusing on job displacement predictions, Malone said that he and the team wanted to explore questions they believe deserve more attention: When do humans and AI work together most effectively? And how can organizations create guidelines and guardrails to ensure these partnerships succeed? The researchers conducted a meta-analysis of 370 results on AI and human combinations in a variety of tasks from 106 different experiments published in relevant academic journals and conference proceedings between January 2020 and June 2023. All the studies compared three different ways of performing tasks: a.) human-only systems b.) AI-only systems, and c.) human-AI collaborations. The overall goal of the meta-analysis was to understand the underlying trends revealed by the combination of the studies. Test Outcomes The researchers found that on average, human-AI teams performed better than humans working alone, but didn't surpass the capabilities of AI systems operating on their own. Importantly, they did not find "human-AI synergy," which means that the average human-AI systems performed worse than the best of humans alone or AI alone on the performance metrics studied. This suggests that using either humans alone or AI systems alone would have been more effective than the human-AI collaborations studied. "There's a prevailing assumption that integrating AI into a process will always help performance -- but we show that that isn't true," said Vaccaro. "In some cases, it's beneficial to leave some tasks solely for humans, and some tasks solely for AI." The team also identified factors affecting how well humans and AI work together. For instance, for decision-making tasks like classifying deep fakes, forecasting demand, and diagnosing medical cases, human-AI teams often underperformed against AI alone. However, for many creative tasks, such as summarizing social media posts, answering questions in a chat, or generating new content and imagery, these collaborations were often better than the best of humans or AI working independently. "Even though AI in recent years has mostly been used to support decision-making by analyzing large amounts of data, some of the most promising opportunities for human-AI combinations now are in supporting the creation of new content, such as text, images, music, and video," said Malone. The team theorized that this advantage in creative endeavors stems from their dual nature: While these tasks require human talents like creativity, knowledge, and insight, they also involve repetitive work where AI excels. Designing an image, for instance, requires both artistic inspiration -- where humans excel -- and detailed execution -- where AI often shines. In a similar vein, writing and generating many kinds of text documents requires human knowledge and insight, but also involves routine, and automated processes such as filling in boilerplate text. "There is a lot of potential in combining humans and AI, but we need to think more critically about it," said Vaccaro. "The effectiveness is not necessarily about the baseline performance of either of them, but about how they work together and complement each other." Optimizing collaboration The research team believes its findings provide guidance and lessons for organizations looking to bring AI into their workplaces more effectively. For starters, Vaccaro emphasized the importance of assessing whether humans and AI are truly outperforming either humans or AI working independently. "Many organizations may be overestimating the effectiveness of their current systems," she added. "They need to get a pulse on how well they're working. Next, they need to evaluate where AI can help workers. The study indicates that AI can be particularly helpful in creative tasks, so organizations should explore what kinds of creative work could be ripe for the insertion of AI. Finally, organizations need to set clear guidelines and establish robust guardrails for AI usage. They might, for example, devise processes that leverage complementary strengths. "Let AI handle the background research, pattern recognition, predictions, and data analysis, while harnessing human skills to spot nuances and apply contextual understanding," Malone suggested. In other words: "Let humans do what they do best." Malone concluded: "As we continue to explore the potential of these collaborations, it's clear that the future lies not just in replacing humans with AI, but also in finding innovative ways for them to work together effectively." Inspired by the increasing use of artificial intelligence (AI) to augment humans, researchers have studied human-AI systems involving different tasks, systems and populations. Despite such a large body of work, we lack a broad conceptual understanding of when combinations of humans and AI are better than either alone. Here we addressed this question by conducting a preregistered systematic review and meta-analysis of 106 experimental studies reporting 370 effect sizes. We searched an interdisciplinary set of databases (the Association for Computing Machinery Digital Library, the Web of Science and the Association for Information Systems eLibrary) for studies published between 1 January 2020 and 30 June 2023. Each study was required to include an original human-participants experiment that evaluated the performance of humans alone, AI alone and human-AI combinations. First, we found that, on average, human-AI combinations performed significantly worse than the best of humans or AI alone (Hedges' g = -0.23; 95% confidence interval, -0.39 to -0.07). Second, we found performance losses in tasks that involved making decisions and significantly greater gains in tasks that involved creating content. Finally, when humans outperformed AI alone, we found performance gains in the combination, but when AI outperformed humans alone, we found losses. Limitations of the evidence assessed here include possible publication bias and variations in the study designs analysed. Overall, these findings highlight the heterogeneity of the effects of human-AI collaboration and point to promising avenues for improving human-AI systems.

When combinations of humans and AI are useful: A systematic review and meta-analysis - Nature Human Behaviour

The remaining moderators we investigated were not statistically significant for human-AI synergy or human augmentation (explanation, confidence, participant type and division of labour). Systems that combine human intelligence and AI tools can address multiple issues of societal importance, from how we diagnose disease to how we design complex systems. But some studies show that augmenting humans with AI can lead to better outcomes than humans or AI working alone, while others show the opposite. These seemingly disparate results raise two important questions: How effective is human-AI collaboration in general? And under what circumstances does this collaboration lead to performance gains versus losses? Our study analyses over three years of recent research to provide insights into both of these questions. Regarding the first question, we found that, on average among recent experiments, human-AI systems did not exhibit synergy: the human-AI groups performed worse than either the human alone or the AI alone. This result complements the qualitative literature reviews on human-AI collaboration, which highlight some of the surprising challenges that arise when integrating human intelligence and AI. For example, people often rely too much on AI systems (overreliance), using its suggestions as strong guidelines without seeking and processing more information. Other times, however, humans rely too little on AI (underreliance), ignoring its suggestions because of adverse attitudes towards automation. Interestingly, we found that, among this same set of experiments, human augmentation did exist in the human-AI systems: the human-AI groups performed better than the humans working alone. Thus, even though the human-AI combinations did not achieve synergy on average, the AI system did on average help humans perform better. This result can occur, of course, because by definition the baseline for human-AI synergy is more stringent than that for human augmentation. It may also occur, however, because obtaining human-AI synergy requires different forms of human-AI interaction, or because the recent empirical studies were not appropriately designed to elicit human-AI synergy. With the large dataset we collected, we also performed analyses of factors that influence the effectiveness of human-AI collaboration. We found that the type of task significantly moderated synergy in human-AI systems: decision tasks were associated with performance losses, and creation tasks were associated with performance gains. We hypothesize that this advantage for creation tasks occurs because even when creation tasks require the use of creativity, knowledge or insight for which humans perform better, they often also involve substantial amounts of somewhat routine generation of additional content that AI can perform as well as or better than humans. For instance, generating a good artistic image usually requires some creative inspiration about what the image should look like, but it also often requires a fair amount of more routine fleshing out of the details of the image. Similarly, generating many kinds of text documents often requires knowledge or insight that humans have and computers do not, but it also often requires filling in boilerplate or routine parts of the text as well. With most of the decision tasks studied in our sample, however, both the human and the AI system make a complete decision, with the humans usually making the final choice. Our results suggest that with these decision tasks, better results might have been obtained if the experimenters had designed processes in which the AI systems did only the parts of the task for which they were clearly better than humans. Only 3 of the 100+ experiments in our analysis explore such processes with a predetermined delegation of separate subtasks to humans and AI. With the four effect sizes from these 3 experiments, we found that, on average, human-AI synergy (g = 0.22, t = 0.69; two-tailed P = 0.494; 95% CI, -0.42 to 0.87) occurred, but the result was not statistically significant (see Supplementary Information section 2.6 for a more detailed discussion of these experiments). Interestingly, when the AI alone outperformed the human alone, substantial performance losses occurred in the human-AI systems. When the human outperformed the AI alone, however, performance gains occurred in the human-AI systems. This finding shows that human-AI performance cannot be explained with a simple average of the human alone and AI alone. In such a case, human-AI synergy could never exist. Most (>95%) of the human-AI systems in our dataset involved humans making the final decisions after receiving input from AI algorithms. In these cases, one potential explanation of our result is that, when the humans are better than the algorithms overall, they are also better at deciding in which cases to trust their own opinions and in which to rely more on the algorithm's opinions. For example, Cabrera et al. used an experimental design in which participants in the human-AI condition saw a problem instance, an AI prediction for that instance and, in some cases, additional descriptions of the accuracy of the AI in this type of instance. The same experimental design, with the same task interface, participant pool and accuracy of the AI system, was used for three separate tasks: fake hotel review detection, satellite image classification and bird image classification. For fake hotel review detection, the researchers found that the AI alone achieved an accuracy of 73%, the human alone achieved an accuracy of 55% and the human-AI system achieved an accuracy of 69%. In this case, we hypothesize that, since the people were less accurate, in general, than the AI algorithms, they were also not good at deciding when to trust the algorithms and when to trust their own judgement, so their participation resulted in lower overall performance than for the AI algorithm alone. In contrast, Cabrera et al. found that, for bird image classification, the AI alone achieved an accuracy of 73%, the human alone achieved an accuracy of 81% and the human-AI system achieved an accuracy of 90%. Here, the humans alone were more accurate than the AI algorithms alone, so we hypothesize that the humans were good at deciding when to trust their own judgements versus those of the algorithms, and the overall performance thus improved over either humans or AI alone. We also investigated other moderators such as the presence of an explanation, the inclusion of the confidence of the AI output and the type of participant evaluated. These factors have received much attention in recent years. Given our result that, on average across our 300+ effect sizes, they do not impact the effectiveness of human-AI collaboration, we think researchers may wish to de-emphasize this line of inquiry and instead shift focus to the significant and less researched moderators we identified: the baseline performance of the human and AI alone, the type of task they perform, and the division of labour between them. We want to highlight some general limitations of our meta-analytic approach to aid with the interpretation of our results. First, our quantitative results apply to the subset of studies we collected through our systematic literature review. To evaluate human-AI synergy, we required that papers report the performance of (1) the human alone, (2) the AI alone and (3) the human-AI system. We can, however, imagine tasks that a human and/or AI cannot perform alone but can when working with the other. Our analysis does not include such studies. Second, we calculated effect sizes that correspond to different quantitative measures such as task accuracy, error and quality. By computing Hedges' g, a unitless standardized effect size, we can describe important relations among these experiments in ways that make them comparable across different study designs with different outcome variables. The studies in our dataset, though, come from different samples of people -- some look at doctors, others at crowdworkers and still others at students -- and this variation can limit the comparability of the effect sizes to a degree. The measurement error can also vary across experiments. For example, some studies estimate overall accuracy on the basis of the evaluation of as many as 500 distinct images, whereas others estimate it on the basis of the evaluation of as few as 15 distinct ones. As is typical for meta-analyses, in our three-level model, we weighted effect sizes as a function of their variance across participants, so we did not account for this other source of variation in measurement. Third, although we did not find evidence of publication biases, it remains possible that they exist, which would impact our literature base and, by extension, our meta-analytic results. However, we expect that if there were a publication bias operating here, it would be a bias to publish studies that showed significant gains from combining humans and AI. And since our overall results showed the opposite, it seems unlikely that they are a result of publication bias. Fourth, our results only apply to the tasks, processes and participant pools that researchers have chosen to study, and these configurations may not be representative of the ways human-AI systems are configured in practical uses of AI outside the laboratory. In other words, even if there is not a publication bias in the studies we analysed, there might be a research topic selection bias at work. Fifth, the quality of our analysis depends on the quality of the studies we synthesized. We tried to control for this issue by only including studies published in peer-reviewed publications, but the rigour of the studies may still vary in degree. For example, studies used different attention check mechanisms and performance incentive structures, which can both affect the quality of responses and thus introduce another source of noise into our data. Finally, we found a high level of heterogeneity among the effect sizes in our analysis. The moderators we investigated account for some of this heterogeneity, but much remains unexplained. We hypothesize that interaction effects exist between the variables we coded (for example, explanation and type of AI), but we do not have enough studies to detect such effects. There are also certainly potential moderators that we did not analyse. For example, researchers mostly used their own experimental platforms and stimuli, which naturally introduce sources of variation between their studies. As the human-AI collaboration literature develops, we hope future work can identify more factors that influence human-AI synergy and assess the interactions among them. Even though our main result suggests that -- on average -- combining humans and AI leads to performance losses, we do not think this means that combining humans and AI is a bad idea. On the contrary, we think it just means that future work needs to focus more specifically on finding effective processes that integrate humans and AI. Our other results suggest promising ways to proceed. In our broad sample of recent experiments, the vast majority (about 85%) of the effect sizes were for decision-making tasks in which participants chose among a predefined set of options. But in these cases we found that the average effect size for human-AI synergy was significantly negative. In contrast, only about 10% of the effect sizes researchers studied were for creation tasks -- those that involved open-ended responses. And in these cases we found that the average effect size for human-AI synergy was positive and significantly greater than that for decision tasks. This result suggests that studying human-AI synergy for creation tasks -- many of which can be done with generative AI -- could be an especially fruitful area for research. Much of the recent work on generative AI with human participants, however, tends to focus on attitudes towards the tool, interviews or think-alouds with participants, or user experience instead of task performance. Furthermore, the relatively little work that does evaluate human-AI collaboration according to quantitative performance metrics tends to report only the performance of the human alone and that of the human-AI combination (not the AI alone). This limitation makes evaluating human-AI synergy difficult, as the AI alone may be able to perform the task at a higher quality and speed than the participants involved in the experiment (typically crowdworkers). We thus need studies that further explore human-AI collaboration across diverse tasks while reporting the performance of the human alone, AI alone and human-AI system. Additionally, as discussed in ref. , human-AI synergy requires that humans be better at some parts of a task, AI be better at other parts of the task and the system as a whole be good at appropriately allocating subtasks to whichever partner is best for that subtask. Sometimes that is done by letting the more capable partner decide how to allocate subtasks, and sometimes it is done by assigning different subtasks a priori to the most capable partner (see Supplementary Information section 2.6 for specific examples from experiments in our dataset). In general, to effectively use AI in practice, it may be just as important to design innovative processes for how to combine humans and AI as it is to design innovative technologies. Many of the experiments in our analysis evaluate performance according to a single measure of overall accuracy, but this measure corresponds to different things depending on the situation, and it omits other important criteria for human-AI systems. For example, as one approaches the upper bound of performance, such as 100% accuracy, the improvements necessary to increase performance usually become more difficult for both humans and AI systems. In these cases, we may wish to consider a metric that applies a nonlinear scaling to the overall classification accuracy and thus takes such considerations into account (Supplementary Information section 1.2). More importantly, there are many practical situations where good performance depends on multiple criteria. For instance, in many high-stakes settings such as radiology diagnoses and bail predictions, relatively rare errors may have extremely high financial or other costs. In these cases, even if AI can, on average, perform a task more accurately and less expensively than humans, it may still be desirable to include humans in the process if the humans are able to reduce the number of rare but very undesirable errors. One potential approach for situations like these is to create composite performance measures that incorporate the expected costs of various kinds of errors. The human augmentation measure described is also appropriate for these high-stakes settings. In general, we encourage researchers to develop, employ and report more robust metrics that consider factors such as task completion time, financial cost and the practical implications of different types of errors. These developments will help us better understand the significance of improvements in task performance as well as the effects of human-AI collaborations. As researchers continue to study human-AI collaboration, we also urge the field to develop a set of commensurability criteria, which can facilitate more systematic comparisons across studies and help us track progress in finding areas of human-AI synergy. These criteria could provide standardized guidelines for key study design elements such as: To further promote commensurability and research synthesis, we encourage the field to establish a standardized and open reporting repository, specifically for human-AI collaboration experiments. This centralized database would host the studies' raw data, code, system outputs, interaction logs and detailed documentation, adhering to the proposed reporting guidelines. It would thus facilitate the replication, extension and synthesis of research in the field. For example, by applying advanced machine learning techniques on such a dataset, we could develop predictive models to guide the design of human-AI systems optimized for specific constraints and contexts. Additionally, it would provide a means to track progress in finding greater areas of human-AI synergy. In conclusion, our results demonstrate that human-AI systems often perform worse than humans alone or AI alone. But our analysis also suggests promising directions for the future development of more effective human-AI systems. We hope that this work will help guide progress in developing such systems and using them to solve some of our most important problems in business, science and society.