Hugging Face Launches HuggingSnap: An Offline AI-Powered Visual Recognition App for iOS

Hugging Face's new iOS app taps AI to describe what you're looking at

AI startup Hugging Face has released a new app for iOS that only does one thing: uses offline, local AI to describe what's in view of your iPhone's camera. The app, called HuggingSnap, taps Hugging Face's in-house vision model, smolvlm2, to analyze what your phone sees in real-time without sending data off to the cloud. Point your camera and ask a question or request a description, and HuggingSnap will identify objects, explain scenes, read text, and generally try to make sense of what you're looking at. The concept certainly isn't novel. Many AI-powered apps -- and Apple's own Apple Intelligence suite of AI-powered features -- can accomplish the same. But as Hugging Face notes in HuggingSnap's App Store description, HuggingSnap works offline, is energy-efficient, and processes all data on your phone. "It's helpful when shopping, traveling, studying, or just exploring your surroundings," the company writes. "HuggingSnap brings smart vision AI to your iPhone." HuggingSnap requires iOS 18 or later. It's also compatible with macOS devices and the Apple Vision Pro, should you wish to use it on a laptop or headset instead.

This AI app claims it can see what I'm looking at - which it mostly can

HuggingSnap is imperfect but demonstrates what can be done entirely on-device. Giving eyesight to AI is becoming increasingly common as tools like ChatGPT, Microsoft Copilot, and Google Gemini roll out glasses for their AI tools. Hugging Face has just dropped its own spin on the idea with a new iOS app called HuggingSnap that offers to look at the world through your iPhone's camera and describe what it sees without ever connecting to the cloud. Think of it like having a personal tour guide who knows how to keep their mouth shut. HuggingSnap runs entirely offline using Hugging Face's in-house vision model, smolVLM2, to enable instant object recognition, scene descriptions, text reading, and general observations about your surroundings without any of your data being sent off into the internet void. That offline capability makes HuggingSnap particularly useful in situations where connectivity is spotty. If you're hiking in the wilderness, traveling abroad without reliable internet, or simply in one of those grocery store aisles where cell service mysteriously disappears, then having the capacity on your phone is a real boon. Plus, the app claims to be super efficient, meaning it won't drain your battery the way cloud-based AI models do. I decided to give the app a whirl. First, I pointed it at my laptop screen while my browser was on my TechRadar biography. At first, the app did a solid job transcribing the text and explaining what it saw. It drifted from reality when it saw the headlines and other details around my bio, however. HuggingSnap thought the references to new computer chips in a headline were an indicator of what's powering my laptop, and seemed to think some of the names in headlines indicated other people who use my laptop. I then pointed my camera at my son's playpen full of toys I hadn't cleaned up yet. Again, the AI did a great job with the broad strokes in describing the play area and the toys inside. It got the colors and even the textures right when identifying stuffed toys versus blocks. It also fell down in some of the details. For instance, it called a bear a dog and seemed to think a stacking ring was a ball. Overall, I'd call HuggingSnap's AI great for describing a scene to a friend but not quite good enough for a police report. HuggingSnap's on-device approach stands out from your iPhone's built-in abilities. While the device can identify plants, copy text from images, and tell you whether that spider on your wall is the kind that should make you relocate, it almost always has to send some information to the cloud. HuggingSnap is notable in a world where most apps want to track everything short of your blood type. That said, Apple is heavily investing in on-device AI for its future iPhones. But for now, if you want privacy with your AI vision, HuggingSnap might be perfect for you.

HuggingSnap app serves Apple's best AI tool, with a convenient twist

Table of Contents Table of Contents It doesn't require internet to work What can you do with HuggingSnap? Machine learning platform, Hugging Face, has released an iOS app that will make sense of the world around you as seen by your iPhone's camera. Just point it at a scene, or click a picture, and it will deploy an AI to describe it, identify objects, perform translation, or pull text-based details. Named HuggingSnap, the app takes a multi-model approach to understanding the scene around you as an input, and it's now available for free on the App Store. It is powered by SmolVLM2, an open AI model that can handle text, image, and video as input formats. Recommended Videos The overarching goal of the app is to let people learn about the objects and scenery around them, including plant and animal recognition. The idea is not too different from Visual Intelligence on iPhones, but HuggingSnap has a crucial leg-up over its Apple rival. Please enable Javascript to view this content It doesn't require internet to work All it needs is an iPhone running iOS 18 and you're good to go. The UI of HuggingSnap is not too different from what you get with Visual Intelligence. But there's a fundamental difference here. Apple relies on ChatGPT for Visual Intelligence to work. That's because Siri is currently not capable of acting like a generative AI tool, such as ChatGPT or Google's Gemini, both of which have their own knowledge bank. Instead, it offloads all such user requests and queries to ChatGPT. That requires an internet connection since ChatGPT can't work in offline mode. HuggingSnap, on the other hand, works just fine. Moreover, an offline approach means no user data ever leaves your phone, which is always a welcome change from a privacy perspective. What can you do with HuggingSnap? HuggingSnap is powered by the SmolVLM2 model developed by Hugging Face. So, what can this model running the show behind this app accomplish? Well, a lot. Aside from answering questions based on what it sees through an iPhone's camera, it can also process images picked from your phone's gallery. For example, show it a picture of any historical monument, and ask it to give you travel suggestions. It can understand the stuff appearing on a graph, or make sense of an electricity bill's picture and answer queries based on the details it has picked up from the document. It has a lightweight architecture and is particularly well-suited for on-device applications of AI. On benchmarks, it performs better than Google's competing open PaliGemma (3B) model and rubs shoulders with Alibaba's rival Qwen AI model with vision capabilities. The biggest advantage is that it requires less system resources to run, which is particularly important in the context of smartphones. Interestingly, the popular VLC media player is also using the same SmolVLM2 model to provide video descriptions, letting users search through a video using natural language prompts. It can also intelligently extract the most important highlight moments from a video. "Designed for efficiency, SmolVLM can answer questions about images, describe visual content, create stories grounded on multiple images, or function as a pure language model without visual inputs," says the app's GitHub repository.