Nvidia Unveils AI Blueprint: Bridging 3D Modeling and AI Image Generation

Nvidia's new tool can turn 3D scenes into AI images

Jess Weatherbed is a news writer focused on creative industries, computing, and internet culture. Jess started her career at TechRadar, covering news and hardware reviews. Nvidia launched a new tool that lets developers generate AI images by first creating them in 3D. The awkwardly named Nvidia AI Blueprint for 3D-guided generative AI is available to download today for computers equipped with an Nvidia RTX 4080 GPU or higher, and works by bridging Blender's 3D modelling software with Black Forest Lab's FLUX.1 image generator. Users can draft a scene in Blender using 3D objects like buildings, plants, animals, and vehicles, which is then used as a reference to create a 2D image. The idea is that manually adjusting the viewing position or where certain objects should be placed will provide more control over finer details when generating 2D images compared to using text descriptions alone. For example, if you have a very specific image of a city in your mind -- the shape and height of the buildings, how many trees or cars are displayed, and even the angle you're viewing it at -- you can use Nvidia's workflow to manually create a rough version of that in Blender. The FLUX.1 image generator will then use that as a base for the images it creates, which could be less frustrating than repeatedly tweaking text-based descriptions until you get something vaguely similar to what you wanted. The 3D objects plotted out in Blender don't even need to be highly detailed, as the tool is only using them as a layout guide. Nvidia describes its blueprints as "pre-defined, customizable AI workflows" that are designed to help developers build generative AI apps. In this case, Nvidia says its AI Blueprint for 3D-guided generative AI provides step-by-step documentation, sample assets, and a preconfigured environment to guide users through using the tool. The workflow isn't entirely unique -- Adobe teased a similar tool during its MAX event in October called "Project Concept" that also lets users map out 3D scenes that can be used to direct image generation. Adobe's variant is still in its experimental stages, however, and may never be released to the wider public.

Control the Composition of AI-Generated Images With the NVIDIA AI Blueprint for 3D-Guided Generative AI

Blueprint combines Black Forest Labs' FLUX.1-dev as an NVIDIA NIM microservice, ComfyUI and Blender in a prebuilt workflow optimized for RTX AI PCs. AI-powered image generation has progressed at a remarkable pace -- from early examples of models creating images of humans with too many fingers to now producing strikingly photorealistic visuals. Even with such leaps, one challenge remains: achieving creative control. Creating scenes using text has gotten easier, no longer requiring complex descriptions -- and models have improved alignment to prompts. But describing finer details like composition, camera angles and object placement with text alone is hard, and making adjustments is even more complex. Advanced workflows using ControlNets -- tools that enhance image generation by providing greater control over the output -- offer solutions, but their setup complexity limits broader accessibility. To help overcome these challenges and fast-track access to advanced AI capabilities, NVIDIA at the CES trade show earlier this year announced the NVIDIA AI Blueprint for 3D-guided generative AI for RTX PCs. This sample workflow includes everything needed to start generating images with full composition control. Users can download the new Blueprint today. Harness 3D to Control AI-Generated Images The NVIDIA AI Blueprint for 3D-guided generative AI controls image generation by using a draft 3D scene in Blender to provide a depth map to the image generator -- FLUX.1-dev, from Black Forest Labs -- which together with a user's prompt generates the desired images. The depth map helps the image model understand where things should be placed. The advantage of this technique is that it doesn't require highly detailed objects or high-quality textures, since they'll be converted to grayscale. And because the scenes are in 3D, users can easily move objects around and change camera angles. Under the hood of the blueprint is ComfyUI, a powerful tool that allows creators to chain generative AI models in interesting ways. For example, the ComfyUI Blender plug-in lets users connect Blender to ComfyUI. Plus, an NVIDIA NIM microservice lets users deploy the FLUX.1-dev model and run it at the best performance on GeForce RTX GPUs, tapping into the NVIDIA TensorRT software development kit and optimized formats like FP4 and FP8. The AI Blueprint for 3D-guided generative AI requires an NVIDIA GeForce RTX 4080 GPU or higher. A Prebuilt Foundation for Generative AI Workflows The blueprint for 3D-guided generative AI includes everything necessary for getting started with an advanced image generation workflow: Blender, ComfyUI, the Blender plug-ins to connect the two, the FLUX.1-dev NIM microservice and the ComfyUI nodes required to run it. For AI artists, it also comes with an installer and detailed deployment instructions. The blueprint offers a structured way to dive into image generation, providing a working pipeline that can be tailored to specific needs. Step-by-step documentation, sample assets and a preconfigured environment provide a solid foundation that makes the creative process more manageable and the results more powerful. For AI developers, the blueprint can act as a foundation for building similar pipelines or expanding existing ones. It comes with source code, sample data, documentation and a working sample for getting started. Real-Time Generation Powered by RTX AI AI Blueprints run on NVIDIA RTX AI PCs and workstations, harnessing recent performance breakthroughs from the NVIDIA Blackwell architecture. The FLUX.1-dev NIM microservice included in the blueprint for 3D-guided generative AI is optimized with TensorRT and quantized to FP4 precision for Blackwell GPUs, enabling more than doubled inference speeds over native PyTorch FP16. For users on NVIDIA Ada Lovelace generation GPUs, the FLUX.1-dev NIM microservice comes with FP8 variants, also accelerated by TensorRT. These improvements make high-performance workflows more accessible for rapid iteration and experimentation. Quantization also helps run models with less VRAM. With FP4, for instance, model sizes are reduced by more than 2x compared with FP16. Customize and Create With RTX AI There are 10 NIM microservices currently available for RTX, supporting use cases spanning image and language generation to speech AI and computer vision -- with more blueprints and services on the way. Available now at https://build.nvidia.com/nvidia/genai-3d-guided, AI Blueprints and NIM microservices provide powerful foundations for those ready to create, customize and push the boundaries of generative AI on RTX PCs and workstations. Each week, the RTX AI Garage blog series features community-driven AI innovations and content for those looking to learn more about NIM microservices and AI Blueprints, as well as building AI agents, creative workflows, digital humans, productivity apps and more on AI PCs and workstations.

New Nvidia AI Blueprint is a bridge between Blender and AI image generation

AI image generation can still be very hit and miss. Even with style and structure references, it can take a lot of guesswork with text prompts and many attempts to get the structure you want for an image. The GPU giant Nvidia thinks it has a solution - and it uses Blender, one of the best 3D modelling software options. The new Nvidia AI Blueprint for 3D-guided generative AI (a catchy moniker, I know) aims to provide more precise control for AI image generation by allowing uses to create scenes in 3D first. It then takes the 3D scenes into Black Forest Lab's Flux.1 image generator. The idea is that you can create a draft layout of an image using Blender, placing things like buildings, plants, vehicles and other objects where you want them. Using Blender means you can adjust the camera position etc. The draft 3D scene will then be used as a reference for the AI model to generate a 2D image. The process provides more control than using text prompts alone because the reference Blender scene can be adjusted visually, reducing the guesswork involved in having to try out multiple prompts to get somewhere near the composition you want. And there's no need for the Blender scene to have a lot of detail or polish because it's only there to provide the composition. Nvidia says the AI Blueprint for 3D-guided generative AI is a ready-to-use workflow that provides an automatic pipeline for intuitive control over image structure, layout, and perspective directly from Blender. The blueprint comes with step-by-step instructions, sample files, and a preconfigured environment, including customisable source code and a demo showing how to integrate ComfyUI workflows. Performance is optimised for RTX AI PCs via Nvidia NIM microservices and accelerated by TensorRT and FP4. Nvidia notes that it's suitable for non-commercial work only. The Nvidia AI blueprint is available for download from the Nvidia website for computers with a Nvidia RTX 4080 GPU or higher.