Google unveils AppFunctions to let Gemini automate tasks across Android apps

Gemini Gets Direct Access to Android Apps Through New Developer Tools

Google announced new developer capabilities that transform how Gemini interacts with Android apps, moving toward what the company calls an "intelligent OS" model. The update introduces AppFunctions, a framework that lets developers expose specific app features directly to AI agents and AI assistants like Gemini1

1

. Google equates AppFunctions to the Model Context Protocol (MCP) used for agents and server-side tools, but these functions execute locally on Android devices rather than remote servers3

3

.

The Samsung Gallery integration on the Samsung Galaxy S26 demonstrates how app automation works in practice. Instead of manually scrolling through albums, users can ask Gemini to "show me pictures of my cat from Samsung Gallery." Gemini processes the request, triggers the relevant function, and displays photos directly within the Gemini app without requiring users to switch applications1

1

. This multimodal experience supports both voice and text input, and retrieved content can be used in follow-up actions like sharing images in messages.

Two Approaches Power the Agentic Future on Android

Google is building this vision on two tracks. AppFunctions provides a structured framework where developers define self-describing functions using the AppFunctions Jetpack library and related platform APIs3

3

. The Gemini app already uses AppFunctions to power integrations with Calendar, Notes, and Tasks across Google apps and OEM defaults1

1

.

For apps without dedicated integrations, Google is developing a UI automation framework that enables AI agents to execute multi-step tasks within installed apps. This system requires zero code from developers, as the platform handles the automation layer2

2

. Users can delegate complex tasks by long-pressing the power button on supported devices. The beta initially supports curated apps in food delivery, grocery, and rideshare categories in the United States and South Korea3

3

.

User Privacy and Control Built Into On-Device Execution

Google designed these developer capabilities with user privacy and security as foundational elements. When Gemini runs tasks through UI automation, users can monitor progress via notifications or a live view, and take manual control if something appears incorrect2

2

. Sensitive actions like purchases trigger alerts before completion. The on-device execution model means actual work happens locally rather than on remote servers3

3

. These safeguards aim to maintain user control while automation runs in the background.

Android 17 Will Expand AI Agent Reach Across Devices

The Samsung Gallery integration is currently available on the Galaxy S26 lineup and will expand to Samsung devices running OneUI 8.5 and higher1

1

. The UI automation preview launches on the Galaxy S26 series and Google Pixel 10 devices as a beta feature within the Gemini app3

3

. Google plans to broaden these capabilities in Android 17 to reach more users, developers, and device manufacturers1

1

. The company is currently working with a limited group of app developers to refine integrations before a broader rollout, with further details expected later this year3

3

. This shift from manually opening apps to telling AI what you need represents a fundamental change in how users interact with their phones.