MULTIMODAL LARGE LANGUAGE MODEL 3D GENERATION

Smart overview of the Invention

Techniques for multimodal large language model (LLM) 3D generation enable efficient creation and modification of 3D objects using natural language. A processing device inputs an object description into a large language model to produce an initial compact graph, which represents a hierarchy of object attributes. This graph is used to generate an initial 3D object model. When an edit is made, an updated compact graph is created, allowing the object model to be modified accordingly. This approach facilitates flexible manipulation of object geometry and materials, enhancing traditional 3D modeling techniques.

Background

Traditional 3D modeling techniques often require manual editing or specialized knowledge, making them time-consuming and less accessible to novice users. These methods typically produce static representations like point clouds and meshes, which are challenging to modify. Advanced techniques such as procedural modeling allow dynamic adjustments but still demand expertise. The limitations of conventional modeling tools can hinder iterative design workflows and reduce productivity, especially when precise modifications are needed across different 3D representations.

System Description

The described system integrates large language models with compact graph representations to generate and edit 3D models. By using natural language instructions, the system bridges the gap between high-level design intent and low-level geometric operations. This integration allows for the creation of editable 3D models that maintain procedural characteristics, improving flexibility and usability for both experienced and novice users. The system's compact graph representation is more efficient in terms of storage and transmission compared to traditional static models.

Implementation

The system operates by first generating an initial compact graph from an object description input into the LLM, which is then used to create a 3D mesh representation. User-provided edit instructions update the compact graph in real-time, enabling rapid iteration and refinement of 3D designs. The system's user interface displays rendered images of the models and provides parameter controls for attribute adjustments, enhancing user interaction and design exploration. This capability is particularly beneficial in industries like entertainment and manufacturing, where iterative design is crucial.

Training and Generalization

To enhance performance, the system employs a multi-stage training process. Initially trained on text descriptions paired with compact graphs, the LLM is fine-tuned using synthetic training data. This involves rendering multi-view images of 3D models, captioning them, and generating text descriptions. This approach improves the system's ability to handle diverse object categories and ensures consistency across 3D modeling tasks. By maintaining multi-view consistency and accurately capturing 3D spatial relationships, the system addresses limitations of traditional 2D-based approaches.