Apple's Matrix3D: A Breakthrough in AI-Powered 3D Scene Generation

New Apple AI model generates 3D scenes from just three images - 9to5Mac

Apple's Machine Learning team, in collaboration with researchers from Nanjing University and The Hong Kong University of Science and Technology, has announced an interesting 3D AI model called Matrix3D. This so-called Large Photogrammetry Model is able to reconstruct 3D objects and scenes from just a few 2D photos, but with a big difference from current pipelines. Here's why this is a big deal. First things first: photogrammetry. It uses photographs to make measurements in order to create 3D models or maps. Currently, this process involves using different models for steps like pose estimation and depth prediction, which can lead to inefficiencies and errors. Matrix3D simplifies this by doing it all in one go. It takes in images, camera parameters (such as angle and focal length), and depth data, and processes them using a unified architecture. This not only simplifies the workflow but also improves accuracy. Even more interesting is how the model was trained. Researchers used a masked learning strategy, very similar to early Transformer-based AI systems that helped pave the way for the first versions of ChatGPT. They randomly hid parts of the input data during the training process, which forced Matrix3D to basically learn how to fill in the gaps. This technique is key because it enables Matrix3D to train effectively even with smaller or incomplete datasets. The results are impressive. With just three input images, Matrix3D can generate detailed 3D reconstructions of objects and even entire environments, which obviously could have very interesting applications for immersive headsets like the Apple Vision Pro.

Apple's New Matrix3D Model Can Turn Flat Images Into Dynamic 3D Scenes

The model was developed in partnership with Nanjing University and HKUST Apple researchers released a new artificial intelligence (AI) model that can generate 3D views from multiple 2D images. The large language model (LLM), dubbed Matrix3D, was developed by the company's Machine Learning team, in collaboration with Nanjing University and the Hong Kong University of Science and Technology (HKUST). The Cupertino-based tech giant has made the AI model available to the open community, and it can be downloaded via Apple's listing on GitHub. With Matrix3D, the researchers have unified the 3D generation pipeline to eliminate the risk of errors. In a post, the tech giant detailed the research that went into the development of the Matrix3D AI model. While several 3D rendering models already exist, this one innovates the existing space by unifying the pipeline to create 3D views. Instead of having multiple models and components, here, a single LLM performs several photogrammetry subtasks such as pose estimation, depth prediction, and novel view synthesis. Notably, Photogrammetry is the technique of obtaining accurate measurements and 3D information about physical objects and environments by analysing images. It is commonly used to create maps, 3D models, and measurements from 2D images taken from different angles. The researchers have also published a paper about the new model on the online preprint journal arXiv. As per the researches, Matrix3D is based on a multimodal diffusion transformer (DiT) architecture. It can integrate data across multiple modalities such as image data, camera parameters, and depth maps. In the paper, Apple researchers highlight that the model was trained using a mask learning strategy where a part of the image is obstructed, and the AI model is trained to find the right pixels that fit in the gap. The researchers found that the LLM can generate an entire 3D object or scene view with just three images from different angles. While the dataset used to train the model was not disclosed, the model itself is available to download, modify, and redistribute via a permissive Apple licence on the company's GitHub listing.