LegoGPT: AI Model Creates Physically Stable Lego Designs from Text Descriptions

New AI model generates buildable Lego creations from text descriptions

On Thursday, researchers at Carnegie Mellon University unveiled LegoGPT, an AI model that creates physically stable Lego structures from text prompts. The new system not only designs Lego models that match text descriptions (prompts) but also ensures they can be built brick by brick in the real world, either by hand or with robotic assistance. "To achieve this, we construct a large-scale, physically stable dataset of LEGO designs, along with their associated captions," the researchers wrote in their paper, which was posted on arXiv, "and train an autoregressive large language model to predict the next brick to add via next-token prediction." This trained model generates Lego designs that match text prompts like "a streamlined, elongated vessel" or "a classic-style car with a prominent front grille." The resulting designs are simple, using just a few brick types to create primitive shapes -- but they stand up. As one Ars Technica staffer joked this morning upon seeing the research, "It builds Lego like it's 1974." In the paper titled "Generating Physically Stable and Buildable Lego Designs from Text," the research team led by Ava Pun explained that many existing 3D generation models focus on making diverse objects with detailed geometry, but these digital designs often can't be physically made. "Without proper support, parts of the design can collapse, float, or remain disconnected," they wrote. Unlike previous attempts at autonomous Lego modeling, LegoGPT reportedly produces step-by-step instructions for building Lego creations that don't fall apart. You can see demos of the system in action on the project's website. How LegoGPT works To build LegoGPT, the Carnegie Mellon team repurposed the technology behind large language models (LLMs), similar to the kind that run ChatGPT, for "next-brick prediction" instead of next-word prediction. To do so, the team fine-tuned LLaMA-3.2-1B-Instruct, an instruction-following language model from Meta. The team then augmented the brick-predicting model with a separate software tool that can verify physical stability using mathematical models that simulate gravity and structural forces. To train the model, the team assembled a new dataset called "StableText2Lego," which contained over 47,000 stable Lego structures paired with descriptive captions generated by a separate AI model, OpenAI's GPT-4o. Each structure underwent physics analysis to ensure it could be built in the real world. LegoGPT works by first generating a sequence of precisely placed Lego bricks. For each new brick in the sequence, the system makes sure it doesn't collide with existing bricks and that it fits within the building space. After completing a design, it uses the aforementioned mathematical models to verify that the model can stand upright without falling apart. If parts would collapse in real life, the system identifies the first unstable brick and backtracks, removing it and all subsequent bricks before trying a different approach. This "physics-aware rollback" method proved essential to the team's approach. Without it, only 24 percent of designs remained standing, compared to 98.8 percent with the full system. The researchers also expanded the system's abilities by adding texture and color options. For example, using an appearance prompt like "Electric guitar in metallic purple," LegoGPT can generate a guitar model, with bricks assigned a purple color. Testing with robots and humans To prove their designs worked in real life, the researchers had robots assemble the AI-created Lego models. They used a dual-robot arm system with force sensors to pick up and place bricks according to the AI-generated instructions. Human testers also built some of the designs by hand, showing that the AI creates genuinely buildable models. "Our experiments show that LegoGPT produces stable, diverse, and aesthetically pleasing Lego designs that align closely with the input text prompts," the team noted in its paper. When tested against other AI systems for 3D creation, LegoGPT stands out through its focus on structural integrity. The team tested against several alternatives, including LLaMA-Mesh and other 3D generation models, and found its approach produced the highest percentage of stable structures. Still, there are some limitations. The current version of LegoGPT only works within a 20×20×20 building space and uses a mere eight standard brick types. "Our method currently supports a fixed set of commonly used Lego bricks," the team acknowledged. "In future work, we plan to expand the brick library to include a broader range of dimensions and brick types, such as slopes and tiles." The researchers also hope to scale up their training dataset to include more objects than the 21 categories currently available. Meanwhile, others can literally build on their work -- the researchers released their dataset, code, and models on their project website and GitHub.

You Can Now Use LegoGPT to Turn Your Text Inputs Into Lego Designs

Certified Sleep Science Coach, Certified Stress Management Coach Ever wanted to take your Lego building game to the next level? A team of computer scientists at Carnegie Mellon University built LegoGPT, the first AI model that takes text inputs and turns them into physically stable Lego designs. Instead of using an AI generator that will churn out a potentially wacky design to fit the request you input, LegoGPT's designs abide by the laws of physics. Read more: The Best Lego Kits According to the team's research, which can be found on GitHub, the AI model was trained on a dataset of over 47,000 Lego structures with 28,000 unique 3D objects. The designs generated with LegoGPT were physically stable 98% of the time, the team said. The tool is free to access on GitHub. You can start by uploading pictures of your existing blocks to determine which building options you have.

You Can Now Use LegoGPT AI to Create Your Own Lego Designs

Certified Sleep Science Coach, Certified Stress Management Coach If you've ever wanted to take your Lego building game to the next level, a team of computer scientists at Carnegie Mellon University has created LegoGPT, the first AI model that takes text inputs and turns them into physically stable Lego designs. Instead of using an AI generator that will churn out a potentially wacky design to fit the request you input, LegoGPT's designs abide by the laws of physics. The AI model was trained on a dataset of over 47,000 Lego structures with 28,000 unique 3D objects, according to the team's research, which can be found on GitHub. The designs generated with LegoGPT are physically stable 98% of the time, the team said. The tool is free to access on GitHub. You can start by uploading pictures of your existing blocks to determine which building options you have.

LegoGPT Eliminates AI Weirdness, Creates Brick Designs You Can Actually Build

A group of computer scientists at Carnegie Mellon University is applying generative AI to Legos with "LegoGPT," which can produce a Lego block design based on the user's text input. "Our experiments show that LegoGPT produces stable, diverse, and aesthetically pleasing Lego designs that align closely with the input text prompts," they wrote in a paper published this week. However, this isn't your average AI image generator, which spit out any image you request. The difference is that LegoGPT creates "stable" 3D Lego designs that respect the laws of physics. The paper notes that most AI-powered 3D object generation doesn't translate into the real world because the "objects may be difficult to assemble or fabricate using standard components" or "the resulting structure may be physically unstable even if assembly is possible." Translation: AI generates some truly strange stuff that you couldn't easily turn into a Lego design. Researchers approached the problem by training the AI model on "physically stable Lego designs paired with captions." This led the team to create a virtual collection, dubbed the StableText2Lego dataset, with 47,000+ Lego structures. Each structure was also run through a program to calculate and assign a "stability score." In addition, LegoGPT can construct physically stable designs by checking for errors during the generation process. "If the resulting design is unstable...we roll back the design to the state before the first unstable brick was generated," the researchers wrote. "We repeat this process iteratively until we reach a stable structure...and continue generation from the partial structure." According to the paper, LegoGPT generated a physically stable design over 98% of the time, outperforming other AI approaches. The program was also smart enough to create valid Lego designs free of errors that adhered to the text prompt 100% of the time. In addition, the researchers say the designs from LegoGPT "can be assembled manually by humans and automatically by robotic arms" in the real world. The researchers uploaded LegoGPT to GitHub, so anyone can download the program and try it out. However, it's currently limited to "20 × 20 × 20 grid" 3D Lego creations. LegoGPT also appears to be focused on simpler designs, rather than the pricey sets the company sells styled after hit properties like Star Wars and Harry Potter. Still, the paper adds: "Our method currently supports a fixed set of commonly used Lego bricks. In future work, we plan to expand the brick library to include a broader range of dimensions and brick types, such as slopes and tiles, allowing for more diverse and intricate Lego designs."

LegoGPT creates Lego designs using AI and text inputs -- tool now available for free to the public

This LLM will unlock the possibilities with your LEGO bricks. A research team from Carnegie Mellon University built an AI model called LegoGPT that outputs valid LEGO designs from text input. According to the team's research paper on GitHub, they trained "an autoregressive large language model to predict the next brick to add via next-token prediction." They also added a validity check and physics-aware rollback during autoregressive inference, ensuring that the final output will always be valid (i.e., no overlapping bricks) and stable (i.e., no floating bricks). Furthermore, LegoGPT's final output can be built by both humans and robots. This is how the team created the dataset -- StableText2Lego -- used to train LegoGPT: a text prompt input is first converted into a ShapeNetCore mesh. This is then plugged into a 20 x 20 x 20 voxel grid from which the initial LEGO brick layout is determined. This layout is then varied while still keeping the overall shape, and then unstable designs are filtered out from the final output. Those left are then rendered in 24 different viewpoints, and then GPT-4o is used to generate descriptions for the final output. The dataset has more than 47,000 LEGO structures that build over 28,000 unique 3D objects, including bookshelves, tables, chairs, cars, ships, guitars, and more. This was then used to train the AI model, allowing it to create unique and original designs solely from text inputs. This is how it creates a new design through text: LegoGPT converts the text into a LEGO design, which is then converted into text tokens ordered from bottom to top. Instructions are then created to pair the structured LEGO bricks with annotations explaining the design, so that the AI will understand the relationships between the text prompt and the physical bricks. From there, LegoGPT predicts the next brick needed to build the design using an autoregressive model. That means it will verify a brick's validity at each step, checking if it is well-formatted, exists in the library, and does not overlap with existing bricks. This will continue until the design is completed, after which its stability is tested. If the AI determines that the output is unstable, it will roll back to the last stable state and continue generating from that point. Once it gets a stable final output, then the design is complete. If you want to play with the AI yourself, the team released its dataset, code, and models, making it easier for anyone to fork the team's work. One development we can see is if someone converts this into a downloadable AI app with a customizable brick library. You can then pair this with a computer vision model or image processing AI. For example, you can take a photo of your available LEGO bricks and let the AI give you a multitude of unique options for building with what you have.

LegoGPT is here to make your blocky dreams come true

At last, an AI model we can really get behind: LegoGPT takes a text prompt and spits out a physically stable design. However, before we get carried away and ask it to come up with a LEGO® Sistine Chapel, the tool can only generate designs that fit within a 20 x 20 x 20 grid using just eight basic brick types (1 x 1, 1 x 2, 1 x 4, 1 x 6, 1 x 8, 2 x 2, 2 x 4, and 2 x 6). There's no "nice part usage" here. What you do get is a model that will take a text prompt and come up with a design that is possible to build using Lego bricks and, crucially, be physically stable. The Carnegie Mellon University research [PDF] comes from Ava Pun, Kangle Deng, Ruixuan Liu, Deva Ramanan, Changliu Liu, and Jun-Yan Zhu. The system works by generating a ShapeNetCore mesh from the entered prompt, then voxelizing it onto a 20 x 20 x 20 grid. "Legolization" then determines the brick layout. "We augment each shape with multiple structural variations by randomizing the brick layout while preserving the overall shape," the team explained. Stability analysis is performed on each variation to remove anything that might fall apart. Researcher Ava Pun told The Register: "We're dreaming of a future where making stuff becomes super personal! Imagine you just type what you want or show us a picture of a chair, and boom - we could actually make that product and ship it to you in just a week or two. "Unfortunately, today's generative AIs cannot offer that. You can generate a cool image or a video of a chair, but the model does not know how things can be made in the real world, like what makes something stable or how parts fit together. "To address this issue, we integrate physical laws and assembly constraints into generative models such as LLMs, enabling us to create objects that function in the real world. We explored this goal in the domain of brick assembly, as it is a widely available medium, and the results can be reproducible across different labs. We believe that our method could be applied to other manufacturing tasks. For example, users with specific ergonomic needs could use it to design custom furniture using a predefined set of parts." While the ambition is to be applauded, and we've seen enough AI-generated imagery to know that real-world physics is all too frequently absent from what gets produced, it's a big jump from something that looks like it came from a childhood box of Lego bricks to shipping a finished product from a text prompt. While acknowledging that "more refinement and human creativity" was needed to make models look impressive rather than blocky approximations, Pun said: "We see it being useful for inspiring new ideas and sketching out initial designs super fast. It could be a neat tool for creators to brainstorm and explore all sorts of different ideas in the early stage." It's certainly a neat tool for brainstorming purposes, one area where AI currently shines. However, with its limited library and grid size, this first iteration is more of an example of what is possible. "In future work, we plan to expand the brick library to include a broader range of dimensions and brick types," Pun said. "Our current system is still quite limited, as it only supports 20 x 20 x 20 dimensions, 20 object categories, and simple brick types. But we are working on expanding the system's capacity. Stay tuned." We asked Lego what it thought of the research and a spokesperson said: "We're unable to comment at this time." ®

Meet LegoGPT, an AI model that creates custom Lego sets

Credit: Ekaterina79 / Editorial RF/ iStock Editorial via Getty Images Researchers at Carnegie Mellon University have unveiled something delightfully geeky: LegoGPT, an AI model that builds Lego structures straight from text prompts. The study, published last Thursday, explains the mechanics in depth. Armed with a massive dataset of Lego builds constructed by the team with captions, the researchers trained a model similar to ChatGPT -- but instead of guessing the next word, it predicts the next brick. It's not the first foray into autonomous Lego construction, but the researchers say LegoGPT stands out by generating step-by-step blueprints designed to keep your builds structurally sound. The team's research, available on GitHub, details how the AI was trained on a dataset of more than 47,000 Lego structures, featuring 28,000 distinct 3D components. According to the researchers, designs generated by LegoGPT were physically stable 98 percent of the time. There's a hefty dose of math and physics behind it all -- more than I can personally vouch for -- but according to the paper, LegoGPT sticks to the laws of physics, so the results aren't especially wild. Most of the team's sample builds were practical pieces: couches, chairs, tables, and similar home designs.

This new AI model can make your dream Lego set - here's how you can try LegoGPT for free

Carnegie Mellon researchers built LegoGPT as open-source and free to try If you've ever stared at a pile of Lego bricks and despaired at making them match the vision in your head, you may be in luck thanks to a new, free AI tool that turns text prompts into real, buildable Lego designs. Describe what you want to build and the aptly named LegoGPT will produce a step-by-step plan using a limited palette of real Lego bricks, with a handy list of which bricks to use and how many you'll need.. To function in the real world, LegoGPT is notably cautious in its approach. While many AI image generators can comfortably spit out wild 3D shapes with zero regard for the laws of physics, LegoGPT runs every design through a literal physics simulator. It checks for weak points. It identifies problem bricks. And if it finds something unstable, it starts all over, reworks the layout, and tries again. It's like how most AI chatbots are a kind of auto-complete for words, hunting for the right one to add to a sentence. Except LegoGPT is predicting the next brick to auto-build a (digital) Lego model. With LegoGPT's answers, you can learn how to turn that colorful plastic pile into brick art. You don't need a PhD in structural engineering or a childhood spent mastering Technic sets, or even the Lego-building robot shown off in a video made by the Carnegie Mellon University researchers behind the new tool. The magic behind LegoGPT comes from a very large dataset called StableText2Lego. The researchers made the dataset by building more than 47,000 stable Lego structures and pairing them with text captions describing their appearance. Rather than spend months or years on that tedious chore, the researchers roped in OpenAI's GPT-4o AI model to analyze rendered images of the Lego structures from 24 different angles and come up with a detailed description they could use. LegoGPT's code, data, and demos are all publicly available on the researchers' website and GitHub. There are some caveats. LegoGPT currently only builds with eight standard brick types, all rectangular, and operates inside a 20-brick cubed space. So you're not getting intricate curved architecture or sprawling castles just yet. Think more early-70s Lego catalog than 4,000-piece Millennium Falcon. Still, the results are fun and very sturdy. The broader implication for generating real-world objects with AI from casual language makes LegoGPT exciting beyond the novelty of making toy blueprints from text descriptions. It promises designs that aren't just possible, but verified to be physically buildable. This could become a cornerstone of prototyping, architectural modeling, and, of course, a weekend activity for Lego hobbyists. But don't dwell too much on the details. You don't need to understand the underlying math to enjoy it. The limitations in size, scope, and brick variety ensure LegoGPT will not replace Lego's in-house designers anytime soon, but it is a leap toward making design more accessible, playful, and connected to the real world. Also, right now, the system doesn't care about color, unless you ask it to. The default focus is purely structural. However, the researchers have already added an optional appearance prompt feature that lets you layer on color schemes. So if you want your electric guitar built in metallic purple, go for it.

LegoGPT can design stable structures using standard LEGOs from text prompts

A team of engineers and AI specialists at Carnegie Mellon University has developed an AI application that can design stable structures from standard LEGOs using text prompts. In their study published on the arXiv preprint server, the group repurposed a large language model (LLM) created by META to build their new system. The research team sought to improve on 3D generative models that currently tend to produce designs that will not work in the real world due to gravity or disconnection issues. To that end, they turned to LEGOs, the brick-like plastic toys that allow children to use their imagination to build structures. To build their system, the team started with META's LLM, LLaMA-3.2-1B-Instruct. To use it to design structures made of bricks, they swapped next-word predictions for next-brick predictions. They also added a separate math-based module that assured physical stability by taking account of structural forces and gravity. Next, they created a training dataset of 47,000 stable LEGO structures and characteristics, with captions generated by another AI system, and then used it to train their AI model. To create a design, the system takes a recursive approach -- bricks are placed and then tested to see if adding them causes instability; if so, the brick is removed and another approach is taken. The researchers tested their system without the rollback feature and found that just 24% of the designs would be stable if built in the real world; with it, the rate went up to 98.8%. The team next added color and texture abilities. To further test their design, the researchers used a pair of robots that could be programmed to build LEGO structures given a design and generated several test structures. They also built some by hand. The team says that their system is capable of producing a wide variety of stable LEGO structures. They also note that they tested their system against other AI systems that have been built to create 3D objects, and found that theirs produced a higher percentage of stable structures.

Researchers Create AI Model That Can Build Physically Stable Lego Designs

LegoGPT is based on a fine-tuned version of LLaMA-3.2-Instruct-1B The AI model was trained on a dataset of over 47,000 Lego structures LegoGPT is an open-source AI model available on GitHub LegoGPT, a new artificial intelligence (AI) model that can generate three-dimensional (3D) Lego structure designs, was unveiled by researchers recently. The new AI model is an open-source project to determine whether AI models can generate structures that are consistent with real-world physics and are consistently stable. The researchers have shared details of how the model was built, as well as have made the dataset available to the public domain. The AI-generated Lego structures were also tested by humans and robots to confirm the stability of the structures. In a post, researchers from Carnegie Mellon University detailed the LegoGPT AI model. The large language model (LLM) can generate a Lego structure from a text prompt, ensuring it is physically stable and buildable. The open-source model is available to download and use on GitHub with a permissive MIT licence. Users can prompt the model to design a "streamline elongated vessel" or a "backless bench with armrest," and it can generate a design that not only matches the description but also can be placed upright without the structure collapsing. This is possible due to two components that make LegoGPT -- the base AI model and a stability analysis system. For the base model, the researchers used a fine-tuned version of the Llama-3.2-Instruct with one billion parameters. This was paired with Gurobi, a mathematical optimisation solver, that runs stability analysis for each generated structure. Alongside building the refined architecture, the researchers also created a dataset to train the model on Lego structures. Dubbed StableText2Lego, it is a dataset containing more than 47,000 Lego structures of over 28,000 unique 3D objects. Each structure is accompanied by detailed captions, design code, and models. To verify that the generated structures are indeed stable, the researchers also tested them with a dual robot assembly. The assembly was tasked with recreating the designs and testing whether they could stand upright. Some of the designs were also recreated by humans to see the impact on stability if less dextrous hands were involved. The research paper claims that 99.8 percent of all structures passed the stability test.