Invention Title:

CAPTURING USER ACTIVITY AS MEMORY FOR QUESTION-ANSWERING

Publication number:

US20260320095

Publication date:
Section:

Physics

Class:

G06F16/334

Inventors:

Assignee:

Applicant:

Smart overview of the Invention

The patent application describes a system designed to enhance question-answering capabilities by capturing and utilizing user activity as memory. This system employs a hierarchical approach to organize and retrieve data, using different levels of indices to store and process information. The core components include processing circuitry and memory that work together to interpret queries, perform semantic searches, and generate answers by leveraging large language models (LLMs).

System Components

The system is structured around a multi-level index framework. It begins with an L2 index that stores event summaries, which are retrieved based on user queries. These summaries form the basis for generating L1 answers, which utilize an L1 index containing screen summaries linked to specific L0 screenshot images. Each level of indices plays a role in refining the information that is passed to the LLM for generating a comprehensive answer to the query.

Functionality

The system captures user activities by taking continuous screenshots during computer usage. These images are processed to generate text descriptions and are organized into events. The descriptions are stored in indices that facilitate searching for relevant screenshots based on user queries. This process allows the system to answer complex questions about past user activities, offering insights and personalized recommendations.

Challenges Addressed

Traditional retrieval-augmented generation (RAG) systems often struggle with complex questions due to limitations in context size and reliance on keyword matching. This system addresses these challenges by enabling reasoning across a sequence of screenshots, allowing for a deeper understanding of user activities. It enhances the ability to provide meaningful answers to abstract or generalized questions by considering broader concepts and user preferences.

Implementation

The implementation involves a computing system composed of various modules that manage different aspects of the memory system. These include modules for generating indices, capturing user profiles, and enhancing RAG capabilities. The system is versatile, capable of being deployed on multiple computing devices, and can automatically or manually capture screenshots. The hierarchical organization of screenshots and their summaries ensures that the LLM can effectively utilize this data for answering questions about historical user activities.