US20260340355
2026-10-29
Physics
G06N3/0475
Pose correlation based digital video generation involves creating digital videos using machine-learning models that enhance realism and coherence, especially when there are significant changes in viewpoints or camera distances. This method utilizes both reference and target pose images to generate a pose correlation map, guiding the diffusion machine-learning model to focus on relevant spatial locations. This approach improves accuracy and reduces visual artifacts, making it more efficient than conventional techniques.
Traditional digital video generation often faces challenges in maintaining temporal coherence between frames, leading to visual artifacts such as jittering and unrealistic movements. These issues are exacerbated when generating videos with significant viewpoint changes or varying camera distances. Conventional methods typically use a single viewpoint and assume a fixed camera-to-object distance, limiting their ability to handle diverse scenarios effectively.
The pose correlation based technique generates a map that quantifies emphasis on regions within feature maps during video generation. This map is derived from reference and target poses, enabling the model to selectively incorporate visual information from multiple reference images. This selective focus enhances the model's ability to generate temporally coherent videos with increased realism, even in complex scenarios involving dramatic viewpoint or distance changes.
The digital video generation system can synthesize frames from a full-body reference image to create digital videos with varying viewpoints and camera distances. This capability supports animations that are "viewpoint-free" and allows for dedicated camera control, unlike conventional methods. The system also addresses the challenge of limited training data by generating datasets with substantial variations in viewpoints and distances, enhancing the model's training and performance.
To train the machine-learning model effectively, the system collects digital videos from sources like conferences and lectures, which feature diverse viewpoints and camera distances. These videos are segmented into clips based on frame similarity and object presence. Filtered clips are annotated for poses, forming the training data. This trained model then implements pose correlation based digital video generation, offering improved video synthesis capabilities.