The Double-Edged Sword of AI: Promise and Pitfalls for Business and Society

AI feels like an unstoppable force. But it is not a panacea for businesses or society

University of Bath provides funding as a member of The Conversation UK. In Greek mythology, Prometheus is credited with giving humans fire as well as the "spark" that spurred civilisation. One of the unintended consequences of Prometheus's "gift" was that the need for celestial Gods diminished. Modern humans have been up to all sorts of things that present similar unintended consequences, from using CFCs that led to a hole in the ozone layer to building systems that they do not understand or cannot fully control. In dabbling with artificial intelligence (AI), humans seem to have taken on the role of Prometheus - apparently gifting machines the "fire" that sparked civilisation. Predicting the future is best left to shamans and futurologists. But we could be better informed about the dangers that follow from how AI operates and work out how to avoid the pitfalls. First, we must recognise that AI holds immense promise for human society. AI is becoming ubiquitous - from mundane tasks such as writing emails to complex settings that require human expertise. AI - by which we mean large language models (LLMs) that appear to "understand" and produce human language - are prediction machines. They are trained on large datasets that enable them to establish statistical associations between a huge number of variables and to predict what is next. If you have used Google, you might have experienced some version of this through its predictive prompts. For example, you might type "how to drive" and Google will complete it as "how to drive an automatic car". It is unlikely to complete it with "how to drive an aeroplane". Google establishes this by looking at the history of what words come after "how to drive". The larger the dataset upon which it has been trained, the more accurate its prediction will be. Variations of this logic are used in all of its current applications. AI's strength, of course, is that it can process untold amounts of data, and extrapolate it to apply to the future. Read more: AI development works better for everyone when its workforce is well looked after But this strength is also its weakness - it makes it vulnerable to a phenomenon management scholars refer to as the "confidence trap". This is the tendency to assume that since earlier decisions have led to positive outcomes, continuing in the same way in future will continue to be OK. Consider an example: the intervals between maintenance of critical aeroplane parts. If increasing the intervals in the past has worked out fine (no failures), these might be adopted widely and there might be a push to increase the intervals further. Yet, it turned out that this was a recipe for disaster. Alaska Airlines flight 261 crashed into the Pacific Ocean killing all 88 people on board because - perhaps influenced by previous success - a decision was made to delay the maintenance of a critical part. AI might just exacerbate this tendency. It can take attention away from signs that there are problems as AI analysis feeds into the picture to inform decision-making. Or AI can extrapolate the results of the past and take decisions without human intervention. Take the example of driverless cars, which have been involved in more than a dozen cases of pedestrians being killed. No dataset, no matter its size, can provide training for every potential action a pedestrian could take. AI cannot yet compete with human discretion in situations like these. But more worryingly, AI can diminish human capabilities to the extent that the ability to determine when to intervene might be lost. Researchers have found that use of AI leads to skill decay - a particular concern where workplace decisions involve life-or-death consequences. Amazon learned the hard way about letting "prediction machines" make decisions when its internal hiring tool discriminated against women as it was trained on a database spanning a ten-year period that skewed towards males. These are, of course, examples that we are aware of. As LLMs get more complex and their inner workings become more opaque, we might not even realise when things go astray. Looking backwards Because AI mirrors the past, it might also be limited in its ability to spark radical innovation. By definition, a radical innovation is a break from the past. Consider the context of photography. Innovative photographers were able to change the way in which the business was done - the history of photojournalism is an example of how something that started as a way of illustrating the news gradually acquired storytelling power and was elevated to the status of an art form. Similarly, fashion designers such as Coco Chanel modernised women's clothing, freeing them from uncomfortable long skirts and corsets that lost their relevance in the post-war world. The founder of sportswear manufacturer Under Armour, former college football player Kevin Plank, used the discomfort from sweaty cotton undershirts as an opportunity to develop clothing using microfibres that draw moisture away from the body. AI can improve on these innovations. But because of how it operates in its current form, it is unlikely to be the source of novelties. Simply put, AI is unable to see or show us the world in a new way, a shortcoming we have termed the "AI Chris Rock problem", inspired by a joke the comedian cracked about making bullets prohibitively expensive. By suggesting a remedy that involved "bullet control" rather than gun control to curb violence, Rock got laughs tapping into the cultural zeitgeist and presenting an innovative solution. In doing so, he also highlighted the absurdity of the situation - something that requires human perception. AI shows its shortcomings when what previously worked loses its relevance or problem-solving power. AI's past success means it will roll out in ever-widening circles - but this itself constitutes a confidence trap that humans should avoid. Prometheus was ultimately rescued by Hercules. No such god stands in the wings for humans. This implies more, rather than less, responsibility rests on our shoulders. Part of this includes ensuring our elected representatives provide regulatory oversight for AI. After all, we cannot let the technocrats play with fire at our expense.

AI feels like an unstoppable force. But it is not a panacea for businesses or society

In Greek mythology, Prometheus is credited with giving humans fire as well as the "spark" that spurred civilization. One of the unintended consequences of Prometheus's "gift" was that the need for celestial Gods diminished. Modern humans have been up to all sorts of things that present similar unintended consequences, from using CFCs that led to a hole in the ozone layer to building systems that they do not understand or cannot fully control. In dabbling with artificial intelligence (AI), humans seem to have taken on the role of Prometheus -- apparently gifting machines the "fire" that sparked civilization. Predicting the future is best left to shamans and futurologists. But we could be better informed about the dangers that follow from how AI operates and work out how to avoid the pitfalls. First, we must recognize that AI holds immense promise for human society. AI is becoming ubiquitous -- from mundane tasks such as writing emails to complex settings that require human expertise. AI -- by which we mean large language models (LLMs) that appear to "understand" and produce human language -- are prediction machines. They are trained on large datasets that enable them to establish statistical associations between a huge number of variables and to predict what is next. If you have used Google, you might have experienced some version of this through its predictive prompts. For example, you might type "how to drive" and Google will complete it as "how to drive an automatic car." It is unlikely to complete it with "how to drive an airplane." Google establishes this by looking at the history of what words come after "how to drive." The larger the dataset upon which it has been trained, the more accurate its prediction will be. Variations of this logic are used in all of its current applications. AI's strength, of course, is that it can process untold amounts of data, and extrapolate it to apply to the future. But this strength is also its weakness -- it makes it vulnerable to a phenomenon management scholars refer to as the "confidence trap." This is the tendency to assume that since earlier decisions have led to positive outcomes, continuing in the same way in future will continue to be OK. Consider an example: the intervals between maintenance of critical airplane parts. If increasing the intervals in the past has worked out fine (no failures), these might be adopted widely and there might be a push to increase the intervals further. Yet, it turned out that this was a recipe for disaster. Alaska Airlines flight 261 crashed into the Pacific Ocean killing all 88 people on board because -- perhaps influenced by previous success -- a decision was made to delay the maintenance of a critical part. AI might just exacerbate this tendency. It can take attention away from signs that there are problems as AI analysis feeds into the picture to inform decision-making. Or AI can extrapolate the results of the past and take decisions without human intervention. Take the example of driverless cars, which have been involved in more than a dozen cases of pedestrians being killed. No dataset, no matter its size, can provide training for every potential action a pedestrian could take. AI cannot yet compete with human discretion in situations like these. But more worryingly, AI can diminish human capabilities to the extent that the ability to determine when to intervene might be lost. Researchers have found that use of AI leads to skill decay -- a particular concern where workplace decisions involve life-or-death consequences. Amazon learned the hard way about letting "prediction machines" make decisions when its internal hiring tool discriminated against women as it was trained on a database spanning a ten-year period that skewed towards males. These are, of course, examples that we are aware of. As LLMs get more complex and their inner workings become more opaque, we might not even realize when things go astray. Looking backwards Because AI mirrors the past, it might also be limited in its ability to spark radical innovation. By definition, a radical innovation is a break from the past. Consider the context of photography. Innovative photographers were able to change the way in which the business was done -- the history of photojournalism is an example of how something that started as a way of illustrating the news gradually acquired storytelling power and was elevated to the status of an art form. Similarly, fashion designers such as Coco Chanel modernized women's clothing, freeing them from uncomfortable long skirts and corsets that lost their relevance in the post-war world. The founder of sportswear manufacturer Under Armour, former college football player Kevin Plank, used the discomfort from sweaty cotton undershirts as an opportunity to develop clothing using microfibres that draw moisture away from the body. AI can improve on these innovations. But because of how it operates in its current form, it is unlikely to be the source of novelties. Simply put, AI is unable to see or show us the world in a new way, a shortcoming we have termed the "AI Chris Rock problem", inspired by a joke the comedian cracked about making bullets prohibitively expensive. By suggesting a remedy that involved "bullet control" rather than gun control to curb violence, Rock got laughs tapping into the cultural zeitgeist and presenting an innovative solution. In doing so, he also highlighted the absurdity of the situation -- something that requires human perception. AI shows its shortcomings when what previously worked loses its relevance or problem-solving power. AI's past success means it will roll out in ever-widening circles -- but this itself constitutes a confidence trap that humans should avoid. Prometheus was ultimately rescued by Hercules. No such god stands in the wings for humans. This implies more, rather than less, responsibility rests on our shoulders. Part of this includes ensuring our elected representatives provide regulatory oversight for AI. After all, we cannot let the technocrats play with fire at our expense.