US20260292096
2026-09-24
Electricity
H04N5/77
The invention focuses on real-time style transformation of video data using neural networks on mobile devices. By applying a trained neural network, native video data from a mobile device's camera is converted into a specific style, such as that of a particular artist or theme. The transformed video is displayed on the mobile device in real-time, maintaining consistency with the camera's current position and orientation. This process enables users to view stylized video data dynamically, similarly to how they would view native live video data.
The invention pertains to image transformation using neural networks, particularly in generating and manipulating multi-view interactive digital media representations. As mobile and wearable devices become more prevalent, there is an increasing demand for digital media formats that go beyond traditional two-dimensional images and videos. Conventional media formats offer limited interactivity, prompting the need for methods that allow live video feeds to be altered and experienced in new ways.
The method involves receiving weighting factors for a neural network on a mobile device, which is trained to convert video images to a specific style. A live video feed from the device's camera, which may change in position and orientation over time, is processed by the neural network to transform the video frames. The transformed video is then output to the device's display in real-time, ensuring that it remains consistent with the camera's movements. This approach allows users to experience a live video feed in a chosen artistic style.
The invention provides mechanisms for modifying live video feeds in real-time, allowing users to view stylized versions of their surroundings. Style transformations can significantly alter the appearance of a live video feed, such as making it look cartoonish or mimic a famous artwork like Van Gogh's "Starry Night." These transformations can be applied selectively, affecting only certain elements within the video, such as people, while leaving other parts unchanged. The invention also discusses the generation of multi-view interactive digital media representations (MVIDMR), which combine image data with location information to create immersive experiences.
The invention explores the use of various data sources, including two-dimensional images, location information, and depth images, to create MVIDMRs. These representations can be enhanced through algorithms for automatic frame selection, stabilization, and view interpolation, improving the user experience. View interpolation, for instance, can render intermediate views to prevent abrupt transitions between frames. Such enhancements can be applied during data capture or post-acquisition, offering flexibility in how MVIDMRs are generated and presented.