MULTIVIEW INPAINTING OF THREE-DIMENSIONAL OBJECTS

Smart overview of the Invention

Techniques for multiview inpainting of three-dimensional (3D) objects are developed to enhance the quality and efficiency of generating inpainted 3D objects. These techniques involve using a machine-learning model to create multiple two-dimensional (2D) views of a 3D object and its mask. Another model reconstructs the inpainted 3D object from these 2D views, which can be displayed in a user interface. This approach addresses limitations of traditional inpainting techniques, which often lack precise control over object placement and size within a 3D model.

Background

Inpainting is a process used in digital 3D modeling to fill or edit regions within a 3D object, supporting tasks like object addition, removal, and artifact correction. Traditional methods often rely on generative techniques that allow users to control the process with simple text prompts. However, these methods typically lack the ability to precisely generate objects in specific locations or sizes within an existing 3D model. This limitation can result in challenges for users who require detailed control over the editing process.

Summary

The described multiview inpainting techniques offer a solution to these challenges by using an inpainting system that generates high-quality 3D objects efficiently. Instead of directly manipulating 3D objects, the system trains a machine-learning model to create 2D images from different perspectives. Another model then reconstructs the 3D object from these images. This method not only reduces computational costs but also supports various 3D representation formats, providing greater user control and flexibility.

Example Application

An example of these techniques is demonstrated with a 3D model of a bear. The system receives the model, a mask indicating where to edit, and a text prompt such as "a bear holding a honey pot." It generates 2D views of the bear and mask, fills the mask with the desired object using a multiview inpainting network, and outputs the 2D views of the edited object. A 3D reconstructor then generates the final 3D model of the bear holding the honey pot, showcasing the system's ability to perform localized edits based on user constraints.

Implementation

The inpainting system is designed to work in various environments and can be implemented on different computing devices, from desktops to mobile devices. It utilizes a service provider system to manage digital services, which are accessible remotely via a network. These services support functionalities such as real-time processing and digital content creation, allowing users to access and use the inpainting techniques effectively. The system's flexibility and efficiency make it a valuable tool for 3D modeling applications across different industries.