ON-DEVICE ABSTRACT AVATAR CREATION AND REAL-TIME ANIMATION FOR EXTENDED REALITY (XR) OR OTHER EXPERIENCES

Smart overview of the Invention

The patent application outlines a method for creating and animating avatars on-device, specifically for extended reality (XR) environments. The process begins with receiving a user input image, which could be an emoji, doodle, or cartoon. Edge detection is used to determine contour lines, followed by ellipse fitting to identify facial features. Landmark points are assigned along these ellipses, which are filtered to refine the facial features. This approach enables real-time, personalized avatar animation without relying on cloud processing.

Image Processing

The method employs a first model to perform edge detection on the user input image, identifying key contour lines. Ellipse fitting is then applied to these lines to determine facial features like eyes and mouth. Unnecessary ellipses are filtered out to retain only significant features. Landmark points are assigned around the remaining ellipses, ensuring accurate representation of facial features for subsequent animation processes.

Audio Integration

Audio frames containing speech data are processed using a second model, which predicts the positions of the landmark points based on audio features. This prediction allows the avatar to exhibit synchronized facial expressions and movements in response to audio input. The integration of audio data ensures that the animated avatar can mimic real-time speech dynamics, enhancing the immersive experience in XR environments.

Avatar Animation

The animated avatar is generated by warping the landmark points according to the predicted positions derived from audio analysis. This process creates a dynamic and responsive avatar that can be used in various applications, including mobile devices, XR devices, and smart displays. The animation is achieved without the need for external processing, facilitating low-latency interactions and preserving user privacy.

Technical Considerations

The invention supports a variety of input images, including objects without facial features, allowing users to select animatable features for overlay. The system is designed to operate efficiently on edge devices, reducing latency and maintaining performance. The method's flexibility and efficiency make it suitable for various applications, from social VR interactions to gaming and smart device interfaces, providing a seamless and personalized user experience.