PERSONALIZED REALISTIC VIDEO GENERATION

Smart overview of the Invention

Personalized realistic video generation involves systems and methods for creating realistic video content tailored to individual users. A client device participates in a video conference by accessing a source video clip and corresponding audio data related to the user. This data is processed by a trained video generator model to produce target video data, which is then streamed during the conference.

Background

The need for personalized video generation arises from the limitations of existing virtual character and text-to-speech models, which often lack naturalness and personalization. These models can be easily identified as artificial, impacting user experience negatively. By employing a trained AI model, personalized video generation aims to overcome these hurdles, providing a more natural and seamless video conferencing experience.

Technical Approach

The process utilizes a generative adversarial network (GAN) to train a personalized video generation engine. The training involves a pre-recorded video of the user, capturing their facial expressions and voice. An autoencoder encodes and decodes both video images and audio, mapping features into a latent space. A discriminator evaluates the authenticity of regenerated content, refining the autoencoder until synthetic content is indistinguishable from real.

Implementation

During use, the user provides an audio clip and a template video to the engine. The engine generates video frames using latent image features and the audio clip, creating a video of the user speaking. This system can be installed on client devices and is adaptable for live communication or on-demand video generation, enhancing user interaction in video conferences.

System Integration

The video generation engine is integrated into a broader videoconferencing system, which includes a chat and video conference provider. This provider connects to client devices through various networks, offering functionalities such as meeting creation, recording, and management. The system supports authentication and authorization services, ensuring secure and personalized user experiences across different videoconferencing scenarios.