Nvidia Expands G-Assist AI with Customizable Plug-in Builder for Enhanced Gaming and PC Experience

Nvidia introduces G-Assist plug-in builder, allowing its AI to integrate with LLMs and software

G-Assist can be modified to perform different actions with different LLMs and software, gaming-related or not. Nvidia is adding customizable plug-ins to Project G-Assist, enabling users to modify Nvidia's AI to suit their needs. The trillion-dollar company has launched a new ChatGPT-based G-Assist Plug-In Builder that allows G-Assist to work with various LLMs and software. The plug-in builder is designed to expand G-Assist's functionality by adding new commands and connecting external tools. "With the plug-in builder, users can generate properly formatted code with AI, then integrate the code into G-Assist -- enabling quick, AI-assisted functionality that responds to text and voice commands." These plug-ins can perform a myriad of functions, from connecting with large language models down to small things such as controlling music. Nvidia states these plug-ins tap into APIs to allow different software and services to talk to each other. Developers can also use coding languages such as JSON and Python to create tools that can be integrated into G-Assist. Nvidia's plug-in builder turns G-Assist from a gaming-focused AI to something that can be tailored for anything else, gaming-related or not. With plug-ins, users can access a responsive small language model that runs on RTX GPUs locally for fast private inference. Nvidia has a GitHub repository that provides users with everything they need to build plug-ins for G-Assist, including instructions and documentation for customizing functionalities. "Developers can define functions in JSON and drop config files into a designated directory, where G-Assist can automatically load and interpret them. Users can even submit plug-ins for review and potential inclusion in the NVIDIA GitHub repository to make new capabilities available for others." Several examples were given of the plug-in's capabilities. One clip showed G-Asist asking Gemini (Google's new AI assistant) for advice on which Legend to pick in Apex Legends. Another showed a developer using a Twitch plug-in that checks if a streamer is live, activating the plug-in through a voice command. Project G-Assist is Nvidia's experimental AI assistant, primarily focused on gaming. The AI is designed to help GeForce gamers troubleshoot performance issues in games, adjust graphics settings, monitor performance, and more. The program is a small language model designed to run locally on RTX GPUs rather than the cloud, improving response times and enabling it to work offline.

Project G-Assist Plug-In Builder Lets Anyone Customize AI on GeForce RTX AI PCs

Extend Project G-Assist System Assistant with easy-to-build AI plug-ins to take action on PCs with natural language. AI is rapidly reshaping what's possible on a PC -- whether for real-time image generation or voice-controlled workflows. As AI capabilities grow, so does their complexity. Tapping into the power of AI can entail navigating a maze of system settings, software and hardware configurations. Enabling users to explore how on-device AI can simplify and enhance the PC experience, Project G-Assist -- an AI assistant that helps tune, control and optimize GeForce RTX systems -- is now available as an experimental feature in the NVIDIA app. Developers can try out AI-powered voice and text commands for tasks like monitoring performance, adjusting settings and interacting with supporting peripherals. Users can even summon other AIs powered by GeForce RTX AI PCs. And it doesn't stop there. For those looking to expand Project G-Assist capabilities in creative ways, the AI supports custom plug-ins. With the new ChatGPT-based G-Assist Plug-In Builder, developers and enthusiasts can create and customize G-Assist's functionality, adding new commands, connecting external tools and building AI workflows tailored to specific needs. With the plug-in builder, users can generate properly formatted code with AI, then integrate the code into G-Assist -- enabling quick, AI-assisted functionality that responds to text and voice commands. Teaching PCs New Tricks: Plug-Ins and APIs Explained Plug-ins are lightweight add-ons that give software new capabilities. G-Assist plug-ins can control music, connect with large language models and much more. Under the hood, these plug-ins tap into application programming interfaces (APIs), which allow different software and services to talk to each other. Developers can define functions in simple JSON formats, write logic in Python and quickly integrate new tools or features into G-Assist. With the G-Assist Plug-In Builder, users can: Built for Builders: Using Free APIs to Expand AI PC Capabilities NVIDIA's GitHub repository provides everything needed to get started on developing with G-Assist -- including sample plug-ins, step-by-step instructions and documentation for building custom functionalities. Developers can define functions in JSON and drop config files into a designated directory, where G-Assist can automatically load and interpret them. Users can even submit plug-ins for review and potential inclusion in the NVIDIA GitHub repository to make new capabilities available for others. Hundreds of free, developer-friendly APIs are available today to extend G-Assist capabilities -- from automating workflows to optimizing PC setups to boosting online shopping. For ideas, find searchable indices of free APIs for use across entertainment, productivity, smart home, hardware and more on publicapis.dev, free-apis.github.io, apilist.fun and APILayer. Available sample plug-ins include Spotify, which enables hands-free music and volume control, and Google Gemini, which allows G-Assist to invoke a much larger cloud-based AI for more complex conversations, brainstorming and web searches using a free Google AI Studio API key. In the clip below, G-Assist asks Gemini for advice on which Legend to pick in the hit game Apex Legends when solo queueing, as well as whether it's wise to jump into Nightmare mode for level 25 in Diablo IV: And in the following clip, a developer uses the new plug-in builder to create a Twitch plug-in for G-Assist that checks if a streamer is live. After generating the necessary JSON manifest and Python files, the developer simply drops them into the G-Assist directory to enable voice commands like, "Hey, Twitch, is [streamer] live?" In addition, users can customize G-Assist to control select peripherals and software applications with simple commands, such as to benchmark or adjust fan speeds, or to change lighting on supported Logitech G, Corsair, MSI and Nanoleaf devices. Other examples include a Stock Checker plug-in that lets users quickly look up real-time stock prices and performance data, or a Weather plug-in allows users to ask G-Assist for current weather conditions in any city. Details on how to build, share and load plug-ins are available on the NVIDIA GitHub repository. Start Building Today With the G-Assist Plugin Builder and open API support, anyone can extend G-Assist to fit their exact needs. Explore the GitHub repository and submit features for review to help shape the next wave of AI-powered PC experiences.

NVIDIA's In-Game Chatbot Can Now Control Spotify and Other External Services

NVIDIA's G-Assist chatbot, a context-aware in-game assistant that provides walkthroughs and graphics setting tweaks, can now interact with external services like Spotify through a new plugin system. It's an odd idea, but it may be useful to those who don't want to open a browser window while gaming. G-Assist is an eight-year old April Fool's joke that became a real piece of software in 2024. The sales pitch is pretty simple -- instead of navigating away from your game to look up walkthroughs, tips, or other info, you can just pull up the G-Assist chatbot and ask it for guidance. Because G-Assist is "context-aware," it can "see" what's going on in your game, and it can automatically adjust your graphics settings or GPU clock speed to improve game performance. Now, thanks to a new plugin builder, developers can add custom functionality to G-Assist. This plugin system is fairly robust, with the ability to make API calls or adjust system controls through C++ and Python bindings. Plugins run locally, of course, and developers can share plugins with other people through GitHub or other repositories. NVIDIA has published a repo of sample plugins, some of which are fairly compelling. There's a plugin that can adjust your RGB lighting, for example, and another that can control Spotify playback. One of NVIDIA's plugins will check if your favorite Twitch streamer is live -- kinda weird, but okay -- while another invokes Google Gemini. I'm not sure why you'd need Gemini while in-game, but I guess it's nice to have the option, as NVIDIA's game walkthrough database is still pretty slim. Community-made plugins will (presumably) be more exciting or useful than what NVIDIA is offering. I also see this as an interesting tool for homelab-obsessed gamers -- you could use G-Assist to interact with apps or services that are running on your local network, such as a self-hosted game server, an LLM, or a game caching server. Of course, plugins designed for local networks may be difficult to share with other people, so developers may focus their attention on web-based services like Spotify and so on. As for whether G-Assist is genuinely "useful" to the average gamer -- ehhh. Even with the plugin system, it's basically just a tool that saves you from navigating away from your games. It might make sense to someone who uses a single-monitor setup (or a gaming laptop). But if you've got more than one screen, hopping into a web browser to look up a walkthrough or open a Spotify playlist isn't much of a hassle. And if you're like me, you rarely multitask or look up walkthroughs while gaming, so an in-game chatbot would mostly go unused. Related Nvidia Drivers on Linux: What You Need to Know Nvidia is known for being somewhat tricky on Linux, but is that reputation earned or exaggerated? Posts 2 I assume that G-Assist will mainly appeal to developers and AI enthusiasts, at least for the time being. New community-made plugins may help the chatbot enter the mainstream, but I don't believe we'll reach that point anytime soon. For what it's worth, Microsoft and Razer are also interested in the idea of context-aware gaming chatbots, so its not like NVIDIA is working in a vacuum -- the industry is pushing for something like G-Assist to gain traction. You can find instructions for loading, building, or sharing plugins at the NVIDIA G-Assist GitHub repository. G-Assist itself is available within the NVIDIA app for Windows and requires a compatible RTX GPU with at least 12GB of VRAM (side note, NVIDIA needs to stop introducing new 8GB GPUs). Note that G-Assist requires about 10GB of storage and may impact game performance. Source: NVIDIA via The Verge

Nvidia's G-Assist AI now works with Twitch, Spotify, and Gemini

Nvidia is releasing plugins for its AI gaming assistant and inviting users to "vibe code" their own with ChatGPT. Nvidia is mighty proud of G-Assist, its new "AI" companion that runs on top of an active game and helps you out with tips or hardware adjustments. Now, it looks like the company is preparing to expand its capabilities. A new plugin-making system and several examples are already letting the tool's tendrils expand to other services. While standard developers can already use the G-Assist API, Nvidia is hoping that more regular users will get in on the action, and has made a Plugin Builder to achieve this. Using ChatGPT's generative code capabilities, it allows even a complete novice to "vibe code" and connect it with anything that also has an open API or a similar setup. The video above shows an example user asking the Plugin Builder tool to generate a plugin to see if a Twitch streamer is live, then walks through the steps to create and install the plugin. It's shockingly smooth, going from a text prompt to a functioning bit of code in two minutes (though I think it's assuming a lot of how-to knowledge on the user's part). Nvidia has already built what it calls "official" G-Assist plugins to hook the system into Twitch, Spotify, Google Gemini, and peripheral and lighting control for Logitech, Corsair, and Nanoleaf, all available on GitHub. Nvidia says it's going to create more plugins in the future, too. Nvidia G-Assist is available in the official Nvidia app now, and naturally it needs an RTX 30-series GPU or later to work.

Anyone can now make plugins for Nvidia's promising G-Assist AI overlay, with options available for Spotify, Twitch, and more

Nvidia G-Assist is one of the few AI tools we in the PC Gamer hardware lair have genuinely liked the look of. G-Assist promises quite a lot, but at bottom it's a local language model you can run off your RTX 30-series (or above) GPU to help you with gaming-related tasks. Now, however, it looks like the number of tasks it'll be able to perform could be about to explode, as it's opening up to community plugin development. Nvidia explains: "This week's AI Garage blog focuses on how Project G-Assist is receiving even more support for tinkering, allowing for lightweight custom plug-ins to improve the PC experience." And further: "Plug-ins are lightweight add-ons that give software new capabilities. G-Assist plug-ins can control music, connect with large language models and much more." Setting aside the questionable use of a dash in the middle of the perfectly respectable word "plugin", this all sounds like very good news for an AI tool that we already liked the look of. Initially it was the ability to have AI help you optimise your game settings that tickled our fancy, but upon its launch last month our James lamented the absence of its gaming AI that can help you make in-game decisions and so on, much like Razer Ava. Throw community-made plugs into the mix and by Jove we might just have an actually impressive and beneficial local AI on our hands. Part of the allure of G-Assist is that it runs in the Nvidia overlay, so no alt-tabbing is necessary. Plus the fact that it's local and can hook in certain data from your system such as, most obviously, info about the game you're playing and its settings. Now, the range of fiddling you'll be able to do via this screen is going to shoot up, as Nvidia says that "Project G-Assist is built for community expansion." Plugin developers will be able to "define functions in JSON and drop config files into a designated directory, where G-Assist can automatically load and interpret them. Users can even submit plug-ins for review and potential inclusion in the NVIDIA GitHub repository to make new capabilities available for others." It looks like this will be able to be achieved with the help of a ChatGPT-based Plugin Builder (yes, that's an AI helping you make plugins for another AI). Instructions are found in the GitHub repo. For the average gamer, there are already some sample plugins which you can drop into your G-Assist folder and use from the get-go. These plugins are "for controlling peripheral & smart home lighting, invoking larger AI models like Gemini, managing Spotify tracks, or even checking streamers' online status on Twitch." The prospect of having lightweight, local plugins of your choice to run off your Nvidia GPU -- rather than trying to, say, cram an entire gigantic LLM onto your local drive -- is very appealing, especially when it can be accessed in-game with an overlay. I, for one, am excited to see what plugins people come up with... you know, so I don't have to do any actual coding myself, with or without the help of an AI.