SUBJECT-AGNOSTIC FACE SWAPPING WITH LOW-RANK ADAPTATION

Smart overview of the Invention

The patent application describes a method for face swapping that leverages low-rank adaptation weights to efficiently swap facial identities between images. The technique involves generating a representation of a first image with a specific facial identity and an identity representation from a second image. This identity representation is then mapped to low-rank adaptation weights used to adapt the first image's representation. The process culminates in decoding these adapted representations to produce an output image where the facial identities are swapped.

Background

Face swapping technology allows for altering the facial identity in images or video frames while maintaining the individual's performance, such as expressions and poses. Traditional methods often require extensive data and training for each subject, which can be resource-intensive. These methods typically involve training models on multiple identities, leading to large memory requirements that increase with each new identity added.

System Overview

The proposed system uses a subject-agnostic model that does not require additional training for each new facial identity. It employs a subject-agnostic fully connected layer with subject-specific weights, minimizing memory usage. The key innovation is using low-rank adaptation weights, which allow for efficient scaling across many identities without needing extensive storage for each one.

System Description

The system comprises a server that includes a model capable of performing face swaps by receiving target and source images. It normalizes these images to align facial features and uses deep learning techniques to identify and swap faces. The model does not store individual parameters for each identity; instead, it calculates identity representations at inference time, mapping them to low-rank adaptation weights that adjust the model's layers accordingly.

Efficiency and Benefits

The use of low-rank adaptation weights reduces the memory footprint and computational resources required for face swapping. This approach eliminates the need to store extensive subject-specific parameters, instead using a compact representation that maintains realistic face swaps. Consequently, the model is more efficient in terms of memory and training data requirements, allowing it to perform effectively across a wide range of facial identities.