Reducing Computational Burden in a Generative Model through Target Vocabulary Constraints

Smart overview of the Invention

The patent application describes a method for efficiently generating responses using a generative model by applying constraints to the target vocabulary. The process begins by encoding a query into a shortlist embedding and a sequence of response-part embeddings. The shortlist embedding is used to identify a reduced-size vocabulary relevant to the query, selected from a larger target vocabulary. This approach limits the number of token possibilities, thereby reducing computational burden and latency in response generation.

The technique involves constructing responses by selecting ranked tokens from the reduced-size vocabulary, based on response-part embeddings. This constraint ensures that only necessary tokens are considered, optimizing the model’s performance. The method can be implemented in two ways: non-autoregressively in a single pass, or autoregressively in multiple passes. The non-autoregressive approach further decreases latency by avoiding iterative processing.

Additionally, the application introduces the use of plural-word tokens, which are tokens comprising multiple words, to capture inter-word dependencies effectively. This is particularly beneficial in the non-autoregressive implementation, as it maintains semantic coherence despite parallel processing. Although the inclusion of plural-word tokens enlarges the target vocabulary, the overall computational load is managed by the imposed vocabulary constraints.

The described technology can be integrated into various systems, devices, and methods, offering flexibility in its application. It is capable of enhancing systems that require real-time query responses by minimizing latency and resource consumption. This adaptability makes it suitable for diverse applications, including digital assistants and search engines, where rapid and accurate response generation is crucial.

Figures included in the patent illustrate the components and processes involved in the response-generating system, such as the RS vocabulary identifier and ordered set generator. These diagrams provide a visual representation of the system's operation, emphasizing the reduction in computational demands through vocabulary constraints. The system's implementation can be supported by various types of computing equipment, as detailed in the later sections of the document.