Grok AI Expands Capabilities with Vision and Multilingual Voice Support
3 Sources
3 Sources
[1]
Even Grok AI Can 'See' Now
There are a lot of trends in generative AI right now. There are the reasoning models like OpenAI's o3, that "think" through each step of a problem before it answers. There are also "deep research" features that can compile information from across the web to generate reports for you. But perhaps the trend that is most "futuristic" of all is Voice Mode. This is the future 2013's Her promised: a chatbot that you can talk to like any other person. The chatbot doesn't say anything differently than it would if you were chatting over text; however, it responds in a "realistic" and "natural" voice, which could create the illusion that you're talking to a person, not a robot. I've never found the feature to be particularly engaging, even from big names like ChatGPT. The tech is impressive, sure, but it's still painfully obvious to my ear that I'm talking to a bot. AI companies haven't been able to shake these identifying quirks, but that hasn't stopped people from forming "relationships" with chatbots -- even falling in love with them. What's more impressive to me is the feature's "vision" component. Some chatbots can not only talk back to you, but can access your camera to see what you're seeing, and incorporate that information in its replies. Both ChatGPT and Gemini offer these features, and now, so does Grok. Grok is the latest chatbot to gain this ability in its Voice Mode. xAI developer Ebby Amir announced the feature, dubbed "Grok Vision," on X Tuesday, noting that Grok Vision supports multilingual audio as well as realtime search. Those latter features are exclusive to SuperGrok subscribers, however. This Tweet is currently unavailable. It might be loading or has been removed. The feature is already live on my end. You can access it by tapping the existing Voice Mode option. If you haven't used this feature already, you'll need to grant Grok permission to access your device's microphone. Following this, you'll be able to start chatting immediately. However, to access Vision, you'll need to tap the camera icon in the bottom left corner. Here, allow Grok to access your camera. Once the feed is live, you can start asking Grok about what it sees. I'm not super keen on sending my live video feed directly to xAI, so I kept my phone directly on the table, so the video feed was all black. Grok, to its credit, tried earnestly to help me fix the problem, suggesting there might be something wrong with the camera, or that my environment was too dark. When I informed it that I had actually taken my phone up to outer space with me, it "laughed," and concluded that had to be the problem: "Ha, outer space, huh? That black feed makes sense now -- no light out there, and the camera's probably not designed for that environment. You might need a space-grade device to get a proper feed." This is the second big feature drop for Grok this month. Last week, xAI rolled out a memory feature for the bot, which allows it to access past conversations for more relevant responses.
[2]
Grok Can Now See Your Surroundings and Speak in Five New Languages
It can identify objects, landmarks, and answer queries about them Grok is getting a couple of new features, the AI firm announced on Wednesday. xAI's artificial intelligence (AI) chatbot is rolling out Grok Vision, a computer vision feature, to its iOS app. Additionally, the company is also shipping support for multilingual audio and real-time web search to the Grok app for iOS and Android smartphones. iPhone users can currently access these features for free, but they are only available to paid subscribers on Android. All of these features are part of the chatbot's Voice Mode. In a post on X (formerly Twitter), Ebby Amir, a member of technical staff at xAI, announced the new Grok features. As mentioned above, all three new features are available to iOS users without a subscription. Android users will need to pay for SuperGrok to access these features. A SuperGrok subscription is priced at Rs. 700 per month and Rs. 6,500 for a year. The biggest addition to the AI chatbot is Grok Vision, which is currently an iOS exclusive feature. Similar to Gemini Live with Video and ChatGPT's Advanced Voice with Video, Grok can now access the device's camera and process the feed in real-time. With this, users can point the device at any object and ask the AI questions about it. Gadgets 360 staff members tested Grok Vision, and the feature seems to have very low latency when connected to a relatively fast Wi-Fi network. In most cases, it was able to correctly identify the object, such as a smartphone or pair of earbuds, or something more abstract like the pattern on a shirt. With multilingual audio support, Grok can now speak in five new languages, alongside English. These include French, Hindi, Japanese, Spanish, and Turkish. While the chatbot previously accepted multilingual text input and generated text in these languages, it can now also understand multilingual verbal prompts and respond in the same language. Additionally, the voice mode is also being upgraded with real-time web search. This means users can ask Grok about current news and other information that requires a web search, and the AI will be able to respond to the queries.
[3]
Elon Musk's Grok AI Can See the World and Talk in Real-Time
Grok Vision is rolling out to iOS users, while only multilingual voice chat is coming to Android users. Elon Musk's xAI has added two new features to its Grok AI chatbot. You can now share your camera with Grok to allow the AI chatbot to see the world around you. xAI is calling it 'Grok Vision' which can see the surroundings and interact with you in different languages. However, there is no option to share your screen yet. In addition to Vision, xAI has added multilingual voice support in Grok so you can voice chat in real-time with Grok, in several local and global languages. xAI is putting effort into making Grok a personal AI chatbot. Recently, Grok received 'Memory' support too which can remember crucial parts from your conversation. Grok Vision is currently rolling out on the Grok app for iOS. Meanwhile, the Grok app for Android gets multilingual audio and real-time search support only. However, you will have to subscribe to the SuperGrok plan which costs $30 per month to access the new features on Android. Lately, many AI labs are starting to offer vision and voice capabilities in their AI chatbots. OpenAI added live audio, screen sharing, and camera sharing in ChatGPT last year in December. Then, Google brought Project Astra to Gemini, which allows the AI chatbot to see the screen and the world around it. Thankfully, Google has made the feature free for all Android users and it's rolling out in a phased manner.
Share
Share
Copy Link
xAI's Grok chatbot now features visual recognition and multilingual voice interaction, enhancing its ability to process real-world information and communicate in various languages.
Grok Vision: A New Frontier in AI Interaction
xAI, the artificial intelligence company founded by Elon Musk, has introduced significant upgrades to its Grok chatbot, positioning it as a more versatile and interactive AI assistant. The latest update brings two major features: Grok Vision and expanded multilingual voice support
1
.Grok Vision allows the AI to access a device's camera, enabling it to process and interpret visual information in real-time. This feature, currently exclusive to iOS users, permits users to point their device at objects and ask questions about them. In testing, Grok Vision demonstrated low latency and accurate object identification, from everyday items to abstract patterns
2
.Multilingual Voice Capabilities
Alongside visual recognition, Grok has expanded its linguistic abilities. The chatbot now supports voice interactions in six languages: English, French, Hindi, Japanese, Spanish, and Turkish. This enhancement allows users to engage in verbal conversations with Grok in their preferred language, with the AI capable of understanding and responding accordingly
2
.Real-Time Web Search Integration
Another notable addition is the integration of real-time web search capabilities within Grok's voice mode. This feature enables users to inquire about current events and up-to-date information, with Grok providing responses based on the latest available data
2
.Availability and Access
The rollout of these features varies across platforms. iOS users can access Grok Vision, multilingual audio support, and real-time web search for free. Android users, however, need to subscribe to the SuperGrok plan, priced at approximately $30 per month, to utilize these advanced features
3
.Competitive Landscape
Grok's new capabilities place it in direct competition with other leading AI chatbots. OpenAI's ChatGPT and Google's Gemini have already introduced similar features, including voice interaction and visual processing. This move by xAI reflects the ongoing trend in the AI industry to create more immersive and versatile chatbot experiences
1
.Related Stories
Implications and User Experience
While these advancements are technologically impressive, some users remain skeptical about the naturalness of AI voice interactions. Despite the improvements, the distinction between human and AI-generated speech remains noticeable to many users. However, this hasn't deterred some individuals from forming emotional connections with AI chatbots
1
.Privacy Considerations
The introduction of camera access raises privacy concerns among users. Some may be hesitant to grant AI systems direct access to their visual surroundings. xAI will need to address these concerns to ensure user trust and adoption of the new features
1
.References
Summarized by
Navi
[1]
[2]
Related Stories
Weekly Highlights
Weekly Highlights
Explore today's top stories
Your Daily Dose of Curated AI News
Don’t drown in AI news. We cut through the noise - filtering, ranking and summarizing the most important AI news, breakthroughs and research daily. Spend less time searching for the latest in AI and get straight to action.
