Google Lens Evolves: AI-Powered Visual Search Expands to Video and Voice

3 ways visual search helps you shop

Google Lens helps people search what they see -- with just a quick photo you're scrolling through visual matches for what caught your eye. Lens is used for nearly 20 billion visual searches every month -- we see shoppers relying on visual search for everything from exploring outfit inspo and living room decor ideas to identifying an item sitting in the background of a video. In fact, today 20 percent of all Google Lens searches are shopping-related. Here are a few ways we're seeing shoppers get the most out of Lens -- plus a handy new update that makes product research with Lens as easy as a snap(shot). In those moments of "What is that? I want it!" Lens merges the power of Google's AI and more than 45 billion products in the Shopping Graph to not only help you discover visual matches of what you love, but also identify specific products and details about them before you buy. Starting today, Lens will prominently display key information when it identifies the product in your photo. Let's say you're at the airport and a backpack someone is wearing catches your eye -- you love how it looks and want to learn more about it. Rather than trying to figure out the exact product name and typing it into Search, you can just tap the Lens icon in the search bar and snap a photo (or upload one from your gallery) to instantly see details like price across retailers, current deals, product reviews and where to buy it -- all in one place, powered by the Shopping Graph.

Google's Visual Search Can Now Answer Even More Complex Questions

Launched in 2017, Google Lens processes 20 billion visual searches a month. Now it will work with video and voice, too. When Google Lens was introduced in 2017, the search feature accomplished a feat that not too long ago would have seemed like the stuff of science fiction: Point your phone's camera at an object and Google Lens can identify it, show some context, maybe even let you buy it. It was a new way of searching, one that didn't involve awkwardly typing out descriptions of things you were seeing in front of you. Lens also demonstrated how Google planned to use its machine learning and AI tools to ensure its search engine shows up on every possible surface. As Google increasingly uses its foundational generative AI models to generate summaries of information in response to text searches, Google Lens' visual search has been evolving, too. And now the company says Lens, which powers around 20 billion searches per month, is going to support even more ways to search, including video and multimodal searches. Another tweak to Lens means even more context for shopping will show up in results. Shopping is, unsurprisingly, one of the key use cases for Lens; Amazon and Pinterest also have visual search tools designed to fuel more buying. Search for your friend's sneakers in the old Google Lens, and you might have been shown a carousel of similar items. In the updated version of Lens, Google says it will show more direct links for purchasing, customer reviews, publisher reviews, and comparative shopping tools. Lens search is now multimodal, a hot word in AI these days, which means people can now search with a combination of video, images, and voice inputs. Instead of pointing their smartphone camera at an object, tapping the focus point on the screen, and waiting for the Lens app to drum up results, users can point the lens and use voice commands at the same time, for example, "What kind of clouds are those?" or "What brand of sneakers are those and where can I buy them?" Lens will also start working over real-time video capture, taking the tool a step beyond identifying objects in still images. If you have a broken record player or see a flashing light on a malfunctioning appliance at home, you could snap a quick video through Lens and, through a generative AI overview, see tips on how to repair the item. First announced at I/O, this feature is considered experimental and is available only to people who have opted into Google's search labs, says Rajan Patel, an 18-year Googler and a cofounder of Lens. The other Google Lens features, voice mode and expanded shopping, are rolling out more broadly. The "video understanding" feature, as Google calls it, is intriguing for a few reasons. While it currently works with video captured in real time, if or when Google expands it to captured videos, entire repositories of videos -- whether in a person's own camera roll or in a gargantuan database like Google -- could potentially become taggable and overwhelmingly shoppable. The second consideration is that this Lens feature shares some characteristics with Google's Project Astra, which is expected to be available later this year. Astra, like Lens, uses multimodal inputs to interpret the world around you through your phone. As part of an Astra demo this spring, the company showed off a pair of prototype smart glasses. Separately, Meta just made a splash with its long-term vision for our augmented reality future, which involves mere mortals wearing dorky glasses that can smartly interpret the world around them and show them holographic interfaces. Google, of course, already tried to realize this future with Google Glass (which uses fundamentally different technology than that of Meta's latest pitch). Are Lens' new features, coupled with Astra, a natural segue to a new kind of smart glasses?