Google Enhances Gemini Live with Visual Guidance and Native Audio Model

Gemini Live is becoming a more helpful digital assistant that can match your mood

The new native audio model can detect nuances in your tone of voice, allowing Gemini to respond in a more natural and empathetic manner. At last year's Made by Google event, Google unveiled Gemini Live, a feature designed for more natural, hands-free conversations with its AI chatbot. Since its debut, Google has introduced numerous upgrades to Gemini Live, most notably the ability to share your camera feed and screen. Today, Google announced a major enhancement to Gemini Live's camera-sharing capabilities and a new audio model to make interactions even more natural.

Gemini Live on Pixel 10 gets native audio, while Pixel Watch 4 lets you raise-to-talk

With the Pixel 10 series and Pixel Watch 4, Google is announcing the latest Gemini app features. When using camera sharing in Gemini Live, the viewfinder can now highlight real-world objects to provide direct guidance. This white rounded rectangle is helpful if you're looking for something in a scene and is like the Project Astra demo at I/O 2025 in May. Google says "real-time visual cues create a powerful new way to learn and solve problems together." Can't decide between two pairs of sneakers? Gemini can see them both and highlight the one that best matches the outfit you're envisioning. Or maybe you're trying to figure out the correct tool to use. Point your camera at your tool box, and Gemini can point out the right one. Visual guidance is launching on the Pixel 10, and will come to other Android devices starting next week. It will be available on iOS in the "coming weeks." In addition to Google Calendar, Keep, and Tasks, Gemini Live will "soon" be able to access Messages, Phone, and Clock, while the Google Maps integration is getting "additional capabilities." Meanwhile, Gemini Live in the "coming weeks" will use a native audio model that can detect if you're excited or concerned and adjust its response accordingly for a more engaging experience. It's a step beyond the current text-to-speech approach. Google says this will "dramatically improve how Gemini Live uses the key elements of human speech, like intonation, rhythm and pitch, allowing for a more responsive and expressive conversation." The Pixel 10 Pro, 10 Pro XL, and 10 Pro Fold comes with one year of Google AI Pro for broader access to the Gemini app like video generation. Meanwhile, the latest version of Gemini Nano powers on-device AI features like Magic Cue, Voice Translate, Pixel Journal, and much more. Today, you can launch Gemini on Wear OS with the "Hey Google" hotword, long-press of the side button, and tapping the watch face complication or Tile. The new gesture lets you just raise your wrist and start talking. This even works if you have disabled tilt-to-wake. The Gemini glow will appear at the bottom of your watch face.

Gemini Live can now coordinate your outfit and remind you - in a calm voice - when it's time to leave

Gemini Live has also expanded its app integrations and introduced expressive voice upgrades Google is continuing its quest to get people to use its Gemini AI assistant at all times and in all places with a new set of upgrades launched alongside the Pixel 10 series of smartphones. The centerpiece of the new and improved Gemini Live is a set of AI eyes, a feature called visual guidance. Basically, you can give Gemini Live access to your camera, and it will look at the same things you're looking at and help you figure out things like the right tool to use, the best choices to coordinate an outfit, or other tasks. The solutions will be right on the screen, with arrows or circles around the correct answer. For now, the feature will only be available on the Pixel 10, but other Android phones and even iOS devices will be able to use the feature in the near future. Visual guidance might sound like a party trick, but it could prove to be a major draw for Gemini Live. Instead of receiving a flat, spoken answer when you ask Gemini to help assemble a new piece of furniture, you can now show the parts to your camera and have the assistant visually indicate which goes where. It doesn't require special hardware; it's like showing a friend who's good at DIY what you have and asking for help. Google clearly sees it as a way to bridge the awkwardness that sometimes happens when you ask an AI for help and it gives you vague or overly generic answers. "Use the blue-handled pliers," might not help much if your toolbox has three tools with blue handles. A glowing circle over the right one is much more helpful. As someone who has tried to follow a YouTube tutorial while simultaneously wielding a screwdriver, I get the appeal. Gemini Live will also sound better when it's showing you things, thanks to new speech models capable of adjusting the tone, and even the character of the voice. So, Gemini might use an especially calm voice to talk about a stressful topic, speed up when you're in a hurry, or perhaps tell you a story about pirates in the stereotypical pirate accent. Gemini Live is also going to be better at multitasking thanks to new links to apps like Google Calendar, Messages, and Maps. So, when you're chatting with Gemini, you could get it to handle your personal appointments and send texts to your friends with directions. The revamp of Gemini Live fits with Google's broader approach to AI, positioning it as an ambient, always-on platform rather than a standalone feature. AI assistance that is flexible enough for any event, while using context to be specifically valuable for individuals, is what Google and other AI developers have promised for a while. And while the visual guidance and other tools aren't going to be perfect, the adaptability could make up for it. You don't need to learn a new system or talk in commands. You just show Gemini what you see, ask it what you need, and get a reply that's tuned to the topic.

Google's Gemini Live AI is finally getting serious about mobile apps

When Google introduced the Gemini Live avatar of its AI, it was nothing short of a revelation. The ability to have freewheeling conversations with an AI, which can also see the world around you through the phone's camera and make sense of it, is something that still astounds me. Now, it's getting ready to work seamlessly with the Android apps installed on your phone. What is changing? At its Pixel 10 launch event, Google announced that Gemini Live can now work with Calendar, Keep, and Tasks. So, let's say you are talking with the AI, and want to quickly save a note, you can do so by simply speaking the task. And while at it, you no longer have to open the Keep app. Recommended Videos While conversing with Gemini Live, you can directly tell it stuff like "save a note about picking up cabbage from the store," or "add an entry for a meeting with John at 4pm this Saturday in Calendar." The idea is to connect Gemini Live seamlessly with apps, just the way Gemini has already been integrated deeply within Google tools such as Docs, Gmail, Search, and more. With Gemini Live, you don't have to click or type, or even switch to another app.. All you need to do is utter the command and it will be seamlessly executed in the background. The best is yet to come Gemini already works with Google's own apps as well as third-party software such as WhatsApp using a system of connectors. Google is now focused on bringing that convenience to its mobile apps via Gemini Live for added voice-first convenience. Aside from Calendar, Keep, and Tasks, Google is also bringing it to the core communication apps, as well. "We're adding the Messages, Phone, and Clock apps and bringing additional capabilities into our Google Maps integration, so Gemini Live will be able to help you in even more ways," the company said in a blog post. Google says it's updating the underlying models to make sure that Gemini Live conversations sound even more natural. Users will also be able to instruct Gemini Live so that it can speak slowly and match their note-taking speed.

Gemini Live: A more helpful, natural and visual assistant

Our vision is for Gemini to be a universal AI assistant you turn to every day -- one that understands the world around you, assists you in completing any task, and is available to explore any idea. At last year's Made by Google, we launched Gemini Live with the Pixel 9 so you can have natural, free-flowing conversations with Gemini. Since then, we've been rapidly improving it by expanding to new languages and countries, bringing the power of your favorite apps into the conversation, and adding the ability to show Gemini what you see by sharing your camera or screen. Today, we're advancing our work to make Gemini an even more helpful assistant with upgrades to Gemini Live -- it's more visually aware, more expressive, and deeply integrated with more of your Google apps. Now, when you share your camera, Gemini not only sees what you see, but it also provides visual guidance by highlighting things directly on your screen. These real-time visual cues create a powerful new way to learn and solve problems together.

Gemini Live can now guide users through parking signs abroad

Google's Gemini Live, initially revealed at last year's Made by Google event, is receiving significant upgrades. These enhancements include visual overlays during camera feed sharing and a new audio model designed for more natural conversations. The upgrades aim to make Gemini Live a more helpful and responsive digital assistant. Since its introduction, Gemini Live has seen several improvements, notably the ability to share camera feeds and screens. Google has now announced an enhancement to its camera-sharing capabilities and a new native audio model to further enhance the naturalness of interactions with the AI chatbot. During the presentation on the forthcoming Google Pixel 10 series, Google provided details regarding upcoming improvements to Gemini Live on Android. A key feature is the addition of visual overlays that highlight specific objects within the camera feed. These visual cues take the form of white-bordered rectangles around the objects of interest, with the surrounding area slightly dimmed to ensure prominence. The "visual guidance" feature is intended to assist users in quickly locating and identifying items within the camera's field of view. Examples of intended uses include highlighting the correct button on a machine, identifying a specific bird within a flock, or pinpointing the right tool for a particular project. The feature also extends to providing advice, such as recommending appropriate footwear for a specific occasion. The visual guidance capability can also manage more challenging scenarios. A Google product manager recounted a personal experience during an international trip where they encountered difficulty interpreting foreign-language parking signs, road markings, and local regulations. Using Gemini Live, the product manager pointed the camera at the scene and inquired about parking permissibility. Gemini Live then consulted local rules, translated the signs, and highlighted an area on the street offering free parking for two hours. Visual guidance will be available directly on the Google Pixel 10 series and will begin its rollout to other Android devices the following week. Expansion to iOS devices is planned in the subsequent weeks. A Google AI Pro or Ultra subscription will not be necessary to access the visual guidance feature. Alongside the visual overlays, Google is implementing a new native audio model within Gemini Live. This model is designed to facilitate more responsive and expressive conversations. The new audio model will respond more appropriately based on the context of the conversation. For instance, when discussing a stressful topic, the audio model will respond using a calmer and more measured tone. Users will have control over the audio model's speech characteristics. If a user finds it difficult to keep up with Gemini's speech, they can request it to speak more slowly. Conversely, when time is limited, users can instruct Gemini to accelerate its speech. The system can also deliver narratives from specific perspectives. As Google stated in its blog post, users can "Ask Gemini to tell you about the Roman empire from the perspective of Julius Caesar himself, and get a rich, engaging narrative complete with character accents." This article was updated at 7:50 PM ET to provide clarifications regarding the natural audio model and incorporate demo assets from Google's blog post.

Gemini Live Will Soon Call and Send Messages on Your Behalf

Google is also working on adding new voice models for the AI assistant Google announced a new update for Gemini Live on Wednesday, adding new capabilities, broader app integration, and improved voice models for more natural speech. The Mountain View-based tech giant is also introducing visual guidance, allowing Gemini Live to highlight specific objects on the screen in response to user queries. In addition, the real-time, two-way voice chatbot is being integrated with the Phone, Messages, and Clock apps, enabling it to make calls, send messages, and set alarms. These features will first roll out to the Google Pixel 10 series. Gemini Live Gets a Major Upgrade In a blog post, the tech giant detailed the new features being added to Gemini Live. Notably, visual guidance will be available on the Pixel 10 series on August 28, the day the smartphones go on sale. The same week, these features will also start rolling out to other Android devices. The iOS devices will receive the feature after that. The company did not mention any specific timeline for the app integration and voice model upgrades. Visual guidance was first teased at this year's Google I/O event. Essentially, when a user turns on the video feed in Gemini Live and the visuals show multiple objects, the AI chatbot can highlight the object it is talking about. On the screen, it appears as a rectangle with white margins. The company says the feature can be useful in scenarios where the visual is crowded by multiple objects, or the user is seeking information about two visually similar objects. Apart from this, the tech giant is also expanding the first-party app integration of Gemini Live. So far, it only supports Google Calendar, Keep and Tasks. However, the company says it is planning to add support for the Phone, Messages, and Clock apps as well. Once the feature rolls out, users can ask the AI to make a call, set an alarm, or send a message to a contact mid-conversation, without needing to stop the conversation. Finally, the company announced that it will soon launch updates to the speech models that power Gemini Live. The update will improve the intonation, rhythm, and pitch to make the audio sound more natural and expressive. Gemini Live will also be able to change the tone of its voice based on the topic, adjust its tempo to speak slower or faster, and even use character accents while narrating a story.

Gemini Live adds visual guidance, expressive speech, conversational photo editing in Google Photos, and more

Google has announced major upgrades to Gemini Live, making the assistant more visually aware, conversational, and connected with Google apps. The update introduces on-screen visual guidance, deeper app integrations, expressive speech capabilities, conversational photo editing, and new transparency features in Google Photos. Gemini Live now offers real-time visual guidance when users share their camera. The assistant can highlight items directly on the screen to help with decision-making and problem-solving. This feature creates a more interactive way to learn and act in real-world scenarios. Gemini Live is expanding support for additional Google apps to help manage everyday activities. For example, while planning a subway route with Gemini, users can quickly ask it to send a message about running late without leaving navigation. Gemini Live will soon deliver more natural and responsive conversations through model updates. Angela Sun, Director of Product Management for the Gemini app, said the new speech model enhances intonation, rhythm, and pitch to create more human-like interactions. Google Photos is gaining a redesigned editor with conversational editing tools powered by Gemini. Users can describe edits naturally without selecting tools or sliders. Selena Shang, Senior Product Manager at Google Photos, said users no longer need to worry about which tools to use, as Gemini can handle both simple tweaks and complex edits through natural language. Google is adding C2PA Content Credentials in the Pixel 10 camera app, covering images taken with or without AI. Within Google Photos, users will be able to view details on how an image was captured or edited. This builds on Google's existing support for IPTC metadata for AI edits and SynthID for Reimagine-generated images, improving transparency around digital content creation.