TOKEN PRUNING BY TEMPORAL PROPAGATION OF ATTENTION ROLLOUT

Smart overview of the Invention

The patent application discusses an electronic device designed for processing sequential input data using a transformer-based neural network model. This device includes a processor and memory that stores the model and instructions for executing specific tasks. The model processes sequential data instances at different time points, generating input tokens that undergo attention-based processing to derive output tokens. Importance scores are calculated for these tokens, which are then used to propagate scores to subsequent data instances, enabling token pruning and reducing computational load.

Background

Transformer-based neural networks are a type of artificial intelligence model used for tasks like detection, recognition, and segmentation. These models process input signals through multiple layers, transforming them into tokens processed with attention mechanisms. Such networks are applied to various data types, including images, text, and audio, especially for sequential data processing. However, the attention mechanism's computational demand increases quadratically with token size, necessitating efficient processing methods.

Innovation

The proposed system enhances efficiency by pruning input tokens based on previous data instances in sequential input data. The process involves calculating attention scores for input tokens, propagating these scores to subsequent instances, and pruning less important tokens. This approach reduces the computational burden of processing all tokens, maintaining the model's effectiveness while improving processing speed and resource utilization.

Technical Details

The device uses attention rollout to determine the influence of input tokens on the neural network's output. Importance scores are generated for each token, reflecting their contribution to the output. These scores are propagated from earlier to later data instances, guiding the pruning of less significant tokens. The pruned tokens are then processed by the attention block, minimizing computational demands without compromising the model's performance.

Illustrations and Implementation

The application includes diagrams illustrating the system architecture and token pruning process. Figures demonstrate attention blocks, token pruning schemes, and the impact on tasks like object detection. A flowchart outlines the sequential data processing steps, showcasing the practical implementation of the described methods. These visual aids support understanding and demonstrate the system's potential applications in AI processing tasks.