AUTO-REGRESSIVE VIDEO GENERATION WITH PAST FRAME CONDITIONING

Smart overview of the Invention

The described technique focuses on generating videos of unlimited length using an autoregressive method that conditions on past frames. This method allows for continuous video generation without temporal inconsistencies, based on user inputs such as text or images. The system iteratively denoises and conditions video frames, integrating features from initial anchor frames with user inputs to maintain consistency across frames.

Background and Challenges

Existing machine learning models for video generation often focus on short sequences, limiting their application in real-world scenarios like advertising or storytelling. Training these models for longer videos is computationally expensive and can lead to quality degradation. The described approach addresses these challenges by using an autoregressive method to generate longer videos while maintaining quality and consistency.

Technical Description

The approach involves conditioning video frame generation on the last frames, using a relationship between data sets to generate conditioning scores. These scores are used as weights to condition the data, often employing attention techniques. This method helps in maintaining scene transitions and object consistency over long video sequences, overcoming typical issues like video stagnation and quality degradation.

Modules for Improved Video Quality

A Conditional Attention Module (CAM) is used to borrow content information from previous frames, ensuring that new frames are not restricted by previous shapes. An Appearance Preservation Module (APM) extracts and maintains object and scene features from an anchor frame, enhancing consistency. These modules, along with high-resolution models and blending techniques, improve the quality and resolution of long videos.

System Components

The Auto-regressive User Input-To-Video System (AUI2VS) includes a Pre-Trained Text-To-Video or Image-To-Video Module that utilizes generative machine learning models. These models generate short video clips from textual descriptions, employing natural language processing and computer vision techniques. The system adapts denoising algorithms for video generation, ensuring alignment with user input content.