Shouldn't there be a way to keep your apps or project data private and improve performance by reducing server latency? This is what on-device AI is designed to solve. It handles AI processing locally, right on your device, without connecting to the internet and sending data to the cloud. In this article, Joas Pambou explains what on-device AI is, why it's important, the tools to build this type of technology, and how it can change the way we use technology every day.
It's not too far-fetched to say AI is a pretty handy tool that we all rely on for everyday tasks. It handles tasks like recognizing faces, understanding or cloning speech, analyzing large data, and creating personalized app experiences, such as music playlists based on your listening habits or workout plans matched to your progress.
But here's the catch:
Where AI tool actually lives and does its work matters a lot.
Take self-driving cars, for example. These types of cars need AI to process data from cameras, sensors, and other inputs to make split-second decisions, such as detecting obstacles or adjusting speed for sharp turns. Now, if all that processing depends on the cloud, network latency connection issues could lead to delayed responses or system failures. That's why the AI should operate directly within the car. This ensures the car responds instantly without needing direct access to the internet.
This is what we call On-Device AI (ODAI). Simply put, ODAI means AI does its job right where you are -- on your phone, your car, or your wearable device, and so on -- without a real need to connect to the cloud or internet in some cases. More precisely, this kind of setup is categorized as Embedded AI (EMAI), where the intelligence is embedded into the device itself.
Okay, I mentioned ODAI and then EMAI as a subset that falls under the umbrella of ODAI. However, EMAI is slightly different from other terms you might come across, such as Edge AI, Web AI, and Cloud AI. So, what's the difference? Here's a quick breakdown:
The main difference? It's about where the AI does the work: on your device, nearby, or somewhere far off in the cloud or web.
On-device AI is, first and foremost, about privacy -- keeping your data secure and under your control. It processes everything directly on your device, avoiding the need to send personal data to external servers (cloud). So, what exactly makes this technology worth using?
On-device AI processes data instantly because it doesn't need to send anything to the cloud. For example, think of a smart doorbell -- it recognizes a visitor's face right away and notifies you. If it had to wait for cloud servers to analyze the image, there'd be a delay, which wouldn't be practical for quick notifications.
Picture this: You are opening an app using voice commands or calling a friend and receiving a summary of the conversation afterward. Your phone processes the audio data locally, and the AI system handles everything directly on your device without the help of external servers. This way, your data stays private, secure, and under your control.
A big win of ODAI is that it doesn't need the internet to work, which means it can function even in areas with poor or no connectivity. You can take modern GPS navigation systems in a car as an example; they give you turn-by-turn directions with no signal, making sure you still get where you need to go.
ODAI AI skips out the round trip of sending data to the cloud and waiting for a response. This means that when you make a change, like adjusting a setting, the device processes the input immediately, making your experience smoother and more responsive.
At its core, ODAI uses special hardware and efficient model designs to carry out tasks directly on devices like smartphones, smartwatches, and Internet of Things (IoT) gadgets. Thanks to the advances in hardware technology, AI can now work locally, especially for tasks requiring AI-specific computer processing, such as the following:
Here's a look at some innovative AI chips in the industry:
These chips or AI accelerators show different ways to make devices more efficient, use less power, and run advanced AI tasks.
Creating AI models that fit resource-constrained devices often requires combining clever hardware utilization with techniques to make models smaller and more efficient. I'd like to cover a few choice examples of how teams are optimizing AI for increased performance using less energy.
Meta's approach to ODAI introduced a model built specifically for smartphones. Instead of scaling traditional models, they designed MobileLLM from scratch to balance efficiency and performance. One key innovation was increasing the number of smaller layers rather than having fewer large ones. This design choice improved the model's accuracy and speed while keeping it lightweight. You can try out the model either on Hugging Face or using vLLM, a library for LLM inference and serving.
This simplifies a model's internal calculations by using lower-precision numbers, such as 8-bit integers, instead of 32-bit floating-point numbers. Quantization significantly reduces memory requirements and computation costs, often with minimal impact on model accuracy.
Neural networks contain many weights (connections between neurons), but not all are crucial. Pruning identifies and removes less important weights, resulting in a smaller, faster model without significant accuracy loss.
Large matrices are a core component of AI models. Matrix decomposition splits these into smaller matrices, reducing computational complexity while approximating the original model's behavior.
This technique involves training a smaller model (the "student") to mimic the outputs of a larger, pre-trained model (the "teacher"). The smaller model learns to replicate the teacher's behavior, achieving similar accuracy while being more efficient. For instance, DistilBERT successfully reduced BERT's size by 40% while retaining 97% of its performance.
Well, all the model compression techniques and specialized chips are cool because they're what make ODAI possible. But what's even more interesting for us as developers is actually putting these tools to work. This section covers some of the key technologies and frameworks that make ODAI accessible.
MediaPipe Solutions is a developer toolkit for adding AI-powered features to apps and devices. It offers cross-platform, customizable tools that are optimized for running AI locally, from real-time video analysis to natural language processing.
At the heart of MediaPipe Solutions is MediaPipe Tasks, a core library that lets developers deploy ML solutions with minimal code. It's designed for platforms like Android, Python, and Web/JavaScript, so you can easily integrate AI into a wide range of applications.
MediaPipe also provides various specialized tasks for different AI needs:
LiteRT or Lite Runtime (previously called TensorFlow Lite) is a lightweight and high-performance runtime designed for ODAI. It supports running pre-trained models or converting TensorFlow, PyTorch, and JAX models to a LiteRT-compatible format using AI Edge tools.
Model Explorer is a visualization tool that helps you analyze machine learning models and graphs. It simplifies the process of preparing these models for on-device AI deployment, letting you understand the structure of your models and fine-tune them for better performance.
You can use Model Explorer locally or in Colab for testing and experimenting.
If you're familiar with PyTorch, ExecuTorch makes it easy to deploy models to mobile, wearables, and edge devices. It's part of the PyTorch Edge ecosystem, which supports building AI experiences for edge devices like embedded systems and microcontrollers.
Gemini is a powerful AI model that doesn't just excel in processing text or images. It can also handle multiple types of data seamlessly. The best part? It's designed to work right on your devices.
For on-device use, there's Gemini Nano, a lightweight version of the model. It's built to perform efficiently while keeping everything private.
Gemini Nano is far from the only language model designed specifically for ODAI. I've collected a few others that are worth mentioning:
Building AI into devices can be exciting and practical, but it's not without its challenges. While you may get a lightweight, private solution for your app, there are a few compromises along the way. Here's a look at some of them:
Phones, wearables, and similar devices don't have the same computing power as larger machines. This means AI models must fit within limited storage and memory while running efficiently. Additionally, running AI can drain the battery, so the models need to be optimized to balance power usage and performance.
AI in devices like drones, self-driving cars, and other similar devices process data quickly, using sensors or lidar to make decisions. However, these models or the system itself don't usually get real-time updates or additional training unless they are connected to the cloud. Without these updates and regular model training, the system may struggle with new situations.
Biases in training data are a common challenge in AI, and ODAI models are no exception. These biases can lead to unfair decisions or errors, like misidentifying people. For ODAI, keeping these models fair and reliable means not only addressing these biases during training but also ensuring the solutions work efficiently within the device's constraints.
These aren't the only challenges of on-device AI. It's still a new and growing technology, and the small number of professionals in the field makes it harder to implement.
Choosing between on-device and cloud-based AI comes down to what your application needs most. Here's a quick comparison to make things clear:
