Google Gemini taps Volvo EX60 cameras to interpret surroundings and control car features

Google's In-Car Gemini Dimmed the Sunroof, Ordered Dinner and Became My Tour Guide

Named a Tech Media Trailblazer by the Consumer Technology Association in 2019, a winner of SPJ NorCal's Excellence in Journalism Awards in 2022 and has three times been a finalist in the LA Press Club's National Arts & Entertainment Journalism Awards. On a balmy, 86-degree day in Mountain View, just outside Google's sprawling campus, I sought refuge from the glaring California sun in a Volvo EX60 -- or so I thought. The air conditioning wasn't working. To mitigate the heat even slightly, we opted to darken the transparent sunroof. And all we had to do was ask Gemini. "Can you make the sunroof opaque?" Vivek Radhakrishnan, a technical program manager at Google, asked the car via voice command. Like magic, the electrochromic window stretching over us blocked the light beaming in from above. We didn't have to rummage through the car's settings to find the right button. At Google's I/O developer conference on Tuesday, I got an early look at a handful of new features coming to cars supporting Google Built-in and Android Auto. The upcoming capabilities, rolling out later this year, are designed to help you keep your eyes on the road while offering helpful information. You can lean on Gemini AI to handle tasks like sharing your ETA, describing that landmark that caught your eye and even helping you order dinner. The Volvo EX60 comes equipped with Google Built-in, a native operating system for car infotainment systems that lets you tap directly into Google's services. We could ask Gemini to identify a dashboard warning light, for example, or have it gauge whether a 65-inch TV we just bought would fit in the back. Although we were technically parked outside of Shoreline Amphitheater, a giant TV in front of us simulated driving along a San Francisco road toward one of the city's signature skyscrapers. We asked Gemini, ""What's that tall building in front of me, and can you tell me something interesting about it?" Using the car's front-facing camera, Gemini identified the Transamerica Pyramid, "which stood as the tallest building in San Francisco for 45 years." We got some bonus information, too. Gemini added, "Nearby, on your right, you'll find the historic copper-clad Sentinel Building, a landmark that miraculously survived the 1906 earthquake and later became home to Francis Ford Coppola's film studio." It was neat to get that much detail and learn something new. Afterwards, we hopped over to the Kia EV9 (which thankfully had functioning A/C) to get a look at upcoming Android Auto features, which are available by connecting your phone to the vehicle. They include a more personalized dashboard design built on Google's Material 3 Expressive, so you can display a picture of your cat alongside custom widgets, for instance. While still parked, we then opened up YouTube and watched videos in 4K resolution at 60 frames per second, which can make the time you spend charging your vehicle a little more entertaining. Once you start driving, those videos automatically shift to audio-only so they're not distracting. Magic Cue can pull information from your email, calendar or other apps to quickly respond to texts asking for your ETA or other event details. In our demo, an incoming text asked, "What time is the pottery class?" and Gemini found the date and time automatically: June 1 at 2 p.m. Google Product Manager Alanna Veiga just had to tap to send it as a reply. If you want to make sure your dinner arrives home when you do, you can also ask Gemini to place an order through a food delivery app. Using voice commands, we asked it to order two fish tacos from Pacific Catch on DoorDash. Gemini then pulled up the DoorDash app on the connected phone and added those items to the cart, tapping through the steps as if it were a human. Once it was time to check out, all Veiga had to do was tap to confirm the order. On vehicles with both Google Built-in and Android Auto compatibility, you'll be able to use the new Immersive Navigation for Google Maps, which shows a 3D view of buildings, overpasses and surrounding terrain, as well as details like lanes, traffic lights and cross walks. That can help you get a better understanding of your surroundings and make navigation clearer and easier. The updates are part of a wider expansion of Gemini into cars, phones, wearables and smart glasses. And while the supercharged AI features might not be able to fix your broken A/C, they can simplify menial tasks and help you stay focused on the road.

The Volvo EX60 Can Literally See The World And Chat With You About It Using Gemini AI

The project has not been released publicly yet, but it looks really promising. A lot of modern cars have at least one camera that can read road signs and help the driver avoid an accident. But a new generation of vehicles is upping the ante, thanks to more advanced computers, crisper sensors, and artificial intelligence. At this year's Google I/O event, Volvo teamed up with the American tech giant to showcase how the Gemini AI assistant that's baked into cars with a Google-based infotainment system can grab information directly from the new EX60's cameras, interpret it in real time, and deliver knowledgeable answers to the driver or passengers. The project is still a work in progress, and the two companies are tight-lipped about when the feature could become widely available, but for an unreleased project, it looks really impressive. The driver can use voice commands to ask the car about a road sign that has words written in another language, and Gemini will automatically scan it, translate it, and describe what it means. As part of the demonstration, the system was asked to describe a tall building that could be seen at the end of the street, and whether or not there was anything to do in or around it. Gemini replied that it was the Transamerica Pyramid in San Francisco, and that it's mainly an office building, so not really a tourist attraction, but there is a park at its base. Additionally, the AI voice assistant asked if it should plan a route to a spot nearby. If you've ever used Google Lens on your smartphone, this is a similar experience, only supercharged with Gemini and natural voice commands. Volvo said that the concept provides an early look at how contextually aware AI experiences will become part of the everyday driving experience in the future. And it's all thanks to the Gemini model's multi-modal understanding, as well as the EX60's neural processing engine and software-defined architecture. Google claims that privacy is at the forefront of this feature -- it's enabled only with the driver's permission, and faces are blurred when images are being analyzed. We don't know when this update will be beamed out to the new Volvo EX60 or other cars with Google built-in, but the tech giant did say that other improvements are coming soon. A car-optimized version of YouTube Music will be released soon for cars with a Google-based infotainment system, along with 3D Immersive Navigation in Google Maps, and a deeper integration for Gemini, which will be able to control various vehicle features through voice commands. Google Maps' Immersive Navigation feature recently debuted on Android smartphones, followed by a rollout on Android Auto, and it's now slated to go live on cars with Google built-in. The first Volvo models to get the new Immersive Navigation feature in Google Maps will be the EX60, EX90, and ES90.

New Google Gemini features means it can "see the world" around you in a Volvo EX60

Through the reveal of the brand new version of Gemini, Google has already unveiled new features and functionality, specifically aimed at drivers. As a first, the upcoming Volvo EX60 will allow Google's Gemini assistant to directly access the car's external cameras and interpret the world around it in real time. Through a special video presentation at Google's I/O conference, as you can see below, you can see that Gemini can use the car's onboard cameras to observe and analyze its surroundings. During Google I/O demos, the system identified San Francisco's Transamerica Pyramid, explained nearby surroundings and translated street signs written in foreign languages. The feature effectively turns the EX60 into a rolling version of Google Lens, only deeply integrated into the vehicle itself through Android Automotive and Volvo's new AI-focused "HuginCore" software architecture. Volvo says the system relies on multimodal AI understanding combined with onboard computing hardware from Qualcomm and Nvidia. Naturally, reactions online have been... mixed. Some users are fascinated by the idea of context-aware AI in cars, while others are already worried about privacy, data collection and overreliance on AI systems for everyday driving tasks. One Reddit commenter bluntly wrote: "How do I disable that?" while another joked that modern cars are becoming "phones on wheels."