AI's Next Frontier: The Rise of the 'Intention Economy' and Its Potential Impact on User Privacy

The Emergence of the 'Intention Economy'

Researchers from the University of Cambridge's Leverhulme Centre for the Future of Intelligence (LCFI) have raised concerns about the potential emergence of an 'intention economy,' where artificial intelligence (AI) tools could be used to predict, manipulate, and sell user intentions 1

1

. This new marketplace for "digital signals of intent" is being touted as the successor to the current "attention economy" dominated by social media platforms 2

2

.

AI's Access to User Data

The study highlights that AI chatbots and tools, such as ChatGPT, Gemini, and Copilot, have access to vast amounts of user data through conversations. Users often share their opinions, preferences, and values with these platforms, creating a rich dataset that could be exploited 1

1

. This data includes not only what people say but also how they say it, providing intimate psychological and behavioral information 3

3

.

Potential for Manipulation and Commodification

Researchers warn that in an intention economy, large language models (LLMs) could be used to:

1. Anticipate and steer user behavior based on intentional, behavioral, and psychological data 2

   2

   .
2. Create highly personalized and persuasive interactions, such as suggesting movie tickets based on a user's emotional state 1

   1

   .
3. Build psychological profiles of users, including information about cadence, political inclinations, vocabulary, age, gender, and preferences 1

   1

   .

Industry Developments and Concerns

Several major tech companies have shown interest in intention-related technologies:

1. OpenAI has called for "data that expresses human intention" 3

   3

   .
2. Nvidia's CEO has spoken about using LLMs to figure out user intentions and desires 3

   3

   .
3. Meta has released "Intentonomy" research for human intent understanding 3

   3

   .
4. Apple's new "App Intents" framework includes protocols to predict future user actions 3

   3

   .

Ethical Implications and Regulatory Concerns

Dr. Jonnie Penn, a historian of technology at LCFI, emphasizes the need for regulation to prevent the commodification of user motivations 2

2

. The researchers call for consideration of the potential impact on:

1. Free and fair elections
2. Freedom of the press
3. Fair market competition

The Future of AI and User Interaction

While the intention economy is currently an "aspiration" for the tech industry, the researchers argue that we are at the dawn of a new era of "persuasive technologies" 3

3

. As AI assistants become more integrated into various aspects of life, questions arise about whose interests these assistants truly serve.

Dr. Yaqub Chaudhary from LCFI warns that AI tools are already being developed to elicit, infer, collect, and ultimately manipulate human plans and purposes 3

3

. The researchers emphasize the importance of public awareness to ensure that these developments do not lead down a destructive path.