The Potential Dark Side of AI: Language Manipulation and Social Control

A 'Newspeak' world - will generative AI help manipulate language for social control?

'Newspeak' in George Orwell's book '1984' was a fictional language designed to limit an individual's ability to think critically. It was intentionally ambiguous and used euphemisms to cloud reality. It eliminated words so people were unable to name concepts and created new ones to support Oceania's, or the state's, political ends. As Orwell said of this terrifying manipulation of language for social control: The Party told you to reject the evidence of your eyes and ears. It was their final, most essential command. But there are growing fears that social media, combined with the increasing use of Large Language Models (LLMs), could take us down a similar path - whether intentionally or not. For starters, UK communications regulator Ofcom pointed out recently that social media was now exposing consumers to less varied information than has previously been the case: Social media platforms expose people to a lot of different news outlets. However, they tend to expose them to a narrower range of news topics than they might encounter on a traditional news website. One concern here is the much debated 'echo chamber' and 'filter bubble' effect. As Pete Wood, Partner and Lead Consultant at cyber-resilience consultancy Naturally Cyber, says: Look at the way social media has already been used by certain actors to consciously influence people's opinions - with conspiracy theories for example. It's about generating emotional responses to hot button topics, based on unconscious biases they already have. But this situation, combined with the danger of both accidental and deliberate data manipulation, is generating concern about where it could all lead. For instance, an academic study conducted by researchers at a number of European universities pointed to a marked drop in the amount of user-generated content on message boards following the introduction of LLMs, such as ChatGPT. The problem here is that such information is the very data ChatGPT is trained on. This implies that over time the quality and quantity of data available for such training will inevitably reduce. The challenge, according to the academics, is that if LLMs use their own self-generated, or synthetic, content to train themselves, it is like "making a photocopy of a photocopy, providing successively less satisfying results". Concerns from other quarters in this context include "model collapse or the introduction of biases" and "intersectional hallucinations". Worryingly, a report by research institute Epoch AI indicated that training data for LLMs could run out as early as 2026 to 2032. Nick Reese, Adjunct Professor at the New York School of Professional Studies' Center for Global Affairs, is not so sure though. He attests that few people are currently using synthetic data - "although it's not zero" - and there is widespread awareness of its problematic nature in the tech sector. However, Reese also acknowledges that training LLMs in this way could take us to a "darker place": LLM companies are already at a place where they've ingested the content of the internet, so they're buying content from publishing houses to get more. There's been debate over the last few years about, 'if we're running out of real data, do we need synthetic data to keep things going?' But synthetic data can create results we can't necessarily predict. So, even in small test cases, it's nearly always less preferable as nuances in the data have small effects that can create larger disparities in outcomes. But there is also the question of why the growing use of LLMs has led to fewer users contributing self-generated content to message boards anyway. The answer, says Wood, is that people are increasingly starting to employ LLMs directly rather than either posting to, or reading, message boards. Sam Raven, AI Risk Consultant at the Risk Crew, agrees: It's partly from convenience, and partly from misplaced trust in the veracity and quality of LLM-generated content...my main concern though is the weakening of the 'critical thinking muscle' and becoming over-reliant on LLMs to think for us. Edward Starkie, Director of Governance, Risk and Compliance at risk consultancy Thomas Murray, is also troubled. He believes people are increasingly relying on LLMs because: They're inherently lazy and if there's a way of interacting that reduces effort, they'll do it. Users adopt LLMs to generate content partly because it's easy. But the problem is that 'bastions of truth' are being undermined through the use of generative AI and mass postings. And fake news is reaching critical mass on platforms, such as social media, which are very difficult to monitor and police. For example, there's been an uptick in the number of negative posts about politicians on the back of real-world events, and that's only possible due to the use of AI. The upshot of this situation, he believes, is: A combination of undermining belief and confidence in the truth, which means it can be more easily influenced by external narratives. Sowing discord and disgruntlement that reflects in the polls is a long-term play, but we're seeing more and more of it. We're also seeing alliances between anti-Western threat actors, who are forming coalitions to amplify the effect of misinformation. Wood agrees: Among the authors and maintainers of systems today, there's a definite intention to prevent LLMs from recommending or generating content that could cause harm. But using them to steer people's opinions is something we should definitely be concerned about. Manipulation of information is the key issue here. It's what we saw certain actors doing in the lead up to Brexit and the US election. Brexit was promoted primarily through the use of big data by Cambridge Analytica, but that was just a precursor to LLMs, which largely automate the process. And that's hugely dangerous as it could impact the future of the human race. While LLM providers claim their products are safe by introducing controls to protect children or prevent physical harm, Wood says: They'll sidestep the issue of influencing people's opinions on social issues. Newspeak is a real threat, undoubtedly, but it was always going to be inevitable when the world embraced social media. So, I'm depressed but not surprised as it plays to people's biases. If people were easily influenced by what we used to call 'tabloid' newspapers, they'll be equally easily misled by social media. And if a feed is fed from LLMs tailored to reinforce those views, the situation will only get worse and worse. Starkie also concurs that generative AI will only make the fake news situation worse over time: It can poison data in the LLM if you put enough information in there to sway the narrative and influence the model. For example, Microsoft realized its Tay chatbot was a bigot after only a few hours as there were no guardrails in place and it took data from internet trolls. The problem here is, he says, that even if information is wrong: It becomes part of the narrative and is referenced again and again until it becomes truth. It's difficult to find the actual truth and trace things back to the genesis of an idea or statement, so most people don't do it. They take things at face value. Independent thinking is difficult when you're bombarded by lots of data and different narratives, and algorithms are designed to do just that. But again, Reese is not convinced: Data manipulation is absolutely a problem with AI - an AI is only as good as the data going in, and you have to be careful what you feed it. But when we're talking about the manipulation of data in LLMs, such as ChatGPT, you have to consider the scale. It's the entire internet plus more. The ability to manipulate data at that scale would be extraordinarily difficult. You can send LLMs fake data on a small scale, which has happened from various message boards including fictional information, and it does cause problems. That's where hallucinations come from. But it would be very difficult to manipulate data at that scale. In the second article in our two-part series, we explore who has most to gain from using generative AI and social media to manipulate data, and how such a situation could play out.

A 'Newspeak' world - if generative AI will be used to manipulate, who stands to benefit?

In this second part of our two-part series, we explore who has the most to gain from using generative AI and social media to manipulate data and how such a situation could play out. Part one looked at the risk to society that generative AI and social media pose in data manipulation terms, and whether it could lead to a 1984 'Newspeak'-based world moving forward. As author and neuroscientist Abhijit Naskaronce said: Whoever controls the narrative, controls the people. One way of doing this in a Large Language Model (LLM) context is to poison data. But another is to include guardrails that limit, or prevent, discussion of certain subjects. Sam Raven, AI Risk Consultant at the Risk Crew, explains: To guard against bias and prejudice, most mainstream LLMs are carefully guard-railed to avoid 'hot topics', which can itself create new biases and blind spots. For example, they won't go anywhere near some religious or some political discussions. It's understandable but the issue is, who gets to decide what's taboo beyond the law? As LLMs amplify particular patterns above others, language can stop referring to the things it's meant to, and pretty soon you have 'Newspeak': euphemism, circumlocution, and the inversion of customary meanings. Moreover, Raven adds, authoritarian regimes, such as China and Russia, have already taken advantage of the fact that the source code for various mainstream LLMs is freely available to develop their own: This means their LLMs are not far behind the West's - and will surely be put to work furthering the manipulation we have already seen these governments wreak using social media in the past. It's the Moon Race all over again, but this time the playing field is much closer to home. And rather than the moon, the goal is 'Strong AI' aka Artificial General Intelligence - which should terrify you. It's terrified [ChatGPT creator] OpenAI so much that it's all over their risk assessments. For example, he says, OpenAI notes that a Strong AI, if connected to the right machinery, could build a weapon of mass destruction. It could also influence users to the extent that their capacity for rational choice could be removed. It is as yet unclear when, or indeed, if Strong AI will emerge though, despite the millions of dollars being pumped into its development by many large players in the industry. But whether or not Strong AI eventually makes an appearance, Raven points out that authoritarian regimes, such as China and Russia, are already in the process of developing their own, third party LLMs: Why? Because it destabilizes societies they are in direct economic and geopolitical competition with...A bad actor, such as Russia, operating an LLM that generated social media content, would certainly have its bots undermining language itself. The Tower of Babel was a great success until those building it could no longer communicate and, therefore, couldn't cooperate. Undermining language undermines society at the most basic level - by eroding the currency of communication. As a result, Raven points out: The AI race is an arms race in many ways, but it's also much more than that. It's now a race for who gets to control the narrative, who gets the super-genius on their team, and who can apply all that to their country's defence, science and economy. As to who stands to benefit most from this situation, Raven attests: Big Tech, of course. Following [President] Trump's election, and his intended rolling back of [President] Biden's AI safeguarding drive, we'll see data monopolies and LLMs used to enforce existing power structures and ideologies...Due to Trump's probable approach and [future head of the Trump regime's Department of Government Efficiency] Elon Musk's robust endorsement of the technology, along with all the potential training data possible from X, I predict a huge amount of privacy issues coming up. The European Union and member states have all signed off on the EU AI Directive, but I'm not sure how much that will change what goes on the behind the very closed doors of training LLMs/GenAI models. Certainly, there will be more landmark copyright cases. Pete Wood, Partner and Lead Consultant at cyber-resilience consultancy Naturally Cyber, takes a similar stance: Rich, influential people don't bat an eyelid at deploying technology to achieve their own personal objectives. So, inevitably, they've expanded from newspapers to social media, which are the newspapers of the 2020s. LLMs allow them to push their opinions onto people without having to drive loads of bots like the Russians did to extract views from Facebook to tailor them for target audiences...If you want to imagine a 'deep state', there isn't one. It's a bunch of rich and powerful individuals who see their task as influencing sheep. The upshot of all this, believes Raven, is that: There will probably be more regulation on AI training data, algorithmic transparency etc to try and prevent monopolies on information. Public sentiment is delicate and could shift after a situation where LLMs create chaos or sow disinformation on a grand scale. But he is concerned that public discourse could also narrow further due to the technology "becoming baked into everyday communication": With info becoming increasingly homogonized, it could impact democracy, education, and even personal freedom...Ultimately, the use of LLMs will be about balancing on the razor's edge of enjoying the benefits of this brave, new world, while avoiding falling down the well of an Orwellian dystopia. But Nick Reese, Adjunct Professor at the New York School of Professional Studies' Center for Global Affairs, takes a different tack: One of the assumptions that underlies all of this is that LLMs will continue to get bigger and bigger and will continue to ingest more and more data. But it's not necessarily true. It's been true since ChatGPT was rolled out as companies sought more and more data to plug into their models. But we're seeing negatives with that, such as hallucinations. So, I'd challenge the underlying assumption that LLMs will get bigger. Instead, we might see them being used for very targeted use cases to answer market needs that current generations of LLMs don't meet. For instance, many government, legal and financial institutions refrain from using general-purpose LLMs as they "can't afford to share confidential data with a multi-party, public cloud". As a result, Reese expects to see the development of smaller, more focused LLMs that organizations use internally: Research, development and innovation with large LLMs will continue. But we'll also see market entrants offering privacy-preserving AIs. It's possible today, but right now the market is somewhat smaller. So, we'll continue to see large LLMs being made available, but there'll be a market reset, in which smaller LLMs will be created and used. But the public perception of what an LLM is will have to change for them to proliferate. Unless national governments find a way to win the support and engagement of LLM platform owners in tackling some of these risks, it is going to be very difficult to do anything about the downsides - as was true in the past of the internet, social media and the like. Governments rarely have the resources or skills to prevent the worst excesses of a free-market economy, and it is always nigh on impossible to put a genie back in its bottle. So, that leaves a good chunk of the responsibility with the tech industry to ensure it does not end up living out its days like Oppenheimer, campaigning against the destructive force it has created. But it is going to take more than just an open letter appealing for collective action in creating responsible AI. It is going to require concrete action - and now.