Adaptive Large Language Model Selection And Refinement For Extracting Data From Documents

Smart overview of the Invention

Adaptive techniques for selecting and deploying language models, particularly large language models (LLMs), are used to extract data from electronic documents. The accuracy of these models is benchmarked by tracking user overrides of extracted data, influencing the choice of model for specific contexts. This selection process considers factors such as accuracy, cost, and latency, allowing for continuous improvement through reinforcement feedback while optimizing for specific target factors.

Adaptive Data Extraction

The adaptive process involves generating prompts for multiple LLMs to extract data from various electronic documents, such as image files, emails, and social media posts. This approach allows for direct comparison of results across different models. Documents are classified by type, such as driver's licenses or medical records, enabling tailored data extraction for specific purposes. The system adapts to different document structures, ensuring relevant information is accurately extracted for data ingestion tasks.

Monitoring and Benchmarking

User interactions with extracted data are monitored to identify overrides, where users correct values populated by LLMs. These corrections, submitted through a user interface, help compute accuracy benchmarks for each model. Benchmarks are based on override frequency, offering insights into model performance. This adaptive mechanism allows for real-time adjustments to model selection without the need for re-coding or retraining, enhancing document processing efficiency.

Refinement of Prompts

To improve accuracy, prompts used for LLMs are refined based on user feedback. Changes in prompt wording can lead to different extraction results, enhancing accuracy for frequently misinterpreted fields. The system dynamically rewrites prompts when accuracy falls below a threshold, selecting the most effective prompts based on override feedback. This refinement process ensures optimal data extraction results across various document types.

AI-Generated Insights

AI-generated insights are provided for data integration workloads, using metrics from the data extraction process. These insights help manage document processing backlogs and identify causes of delays. By forming prompts with specific processing metrics and guidelines, the system extracts valuable insights from LLM outputs, aiding in efficient document management and processing.