US20260278899
2026-09-17
Physics
G06T13/205
The patent application describes a system for audio-driven character animation using diffusion models. These models are trained to improve the realism and accuracy of character animations in real-time or near real-time applications. The diffusion model can encode and decode various facial components separately, such as the tongue, eyes, and gums, to enhance the animation's lifelike quality. It processes audio segments in stages, storing state information to ensure consistency across different processing phases. This approach reduces latency by limiting the number of diffusion steps needed to generate denoised frames of the animated character.
Interactive characters, like avatars, often rely on models that render animations based on audio inputs. Traditional regression-based models can lead to inaccuracies, averaging multiple output variations into a final animation that may not accurately reflect the audio input. This can result in animations with suppressed facial expressions or imprecise lip syncing. Additionally, these models often require processing the entire audio input before generating frames, which is inefficient for real-time applications. The diffusion model proposed addresses these limitations by offering more precise and realistic animations.
The diffusion model improves character animation by modeling the distribution of possible outputs for each time step, unlike conventional regression models that produce a single output. This allows for more realistic facial expressions and mouth shapes, as a given sound can correspond to multiple animations. The model processes audio using fewer diffusion steps and segmented audio portions, reducing latency. This makes it suitable for real-time applications, enhancing the responsiveness and realism of animated characters.
The system uses a diffusion model with an advanced architecture for processing inputs to create animated character outputs. Inputs include audio, noisy frames, timestamps, subject identifiers, and emotional states. The model processes these inputs to generate denoised frames that align with the audio. It utilizes encoders to create features from the inputs, which are then processed through layers like GRUs, RNNs, or transformers. Decoders with multiple heads handle different facial components, combining outputs to form the animation. This method allows the model to predict denoised animations, enhancing motion realism.
The system is configured to minimize latency, enabling its use in real-time applications. By reducing the number of diffusion steps and segmenting audio processing, the model efficiently generates animations. It processes audio in stages, allowing for continuous output of denoised frames. This staged processing uses state data from previous inputs to maintain consistency across frames. The model's ability to handle audio in segments ensures it can produce real-time animations without processing the entire audio input at once, making it ideal for interactive applications.