Invention Title:

VISION FINE-TUNING FOR MACHINE LEARNING MODEL AND APPLICATION PROGRAMMING INTERFACE (API)

Publication number:

US20260311048

Publication date:
Section:

Physics

Class:

G06N20/00

Inventors:

Assignee:

Applicant:

Smart overview of the Invention

The technology focuses on enhancing generative response engines by enabling them to process multi-modal inputs, including both text and images. This advancement is achieved through a fine-tuning process that utilizes a training set received via an API call. The training set comprises input images linked to text and output text for supervised learning. The API facilitates the fine-tuning of a base model, enabling it to generate responses based on multi-modal inputs more effectively.

Background

Generative response engines, such as large language models, have transformed artificial intelligence by advancing natural language understanding and generation. These engines excel in text-related tasks but traditionally rely on publicly available data. The integration of multi-modal inputs, such as images and text, represents a new frontier, allowing models to perform complex tasks by combining information from various sources. This multi-modal approach aims to enhance the engines' capabilities beyond single-modality systems.

Fine-Tuning Process

The fine-tuning process involves using an API and a user interface to upload training sets containing multi-modal inputs. This method allows users to tailor a base model for specific tasks, such as visual search or medical image analysis. Vision fine-tuning, for instance, enhances a model's ability to interpret images, significantly improving its performance. The process is efficient, requiring as few as 100 images to achieve notable enhancements, with greater improvements possible with larger datasets.

Model Architecture

The architecture of multi-modal machine learning models includes tokenization layers that convert image and text inputs into tokens. These tokens are processed through embedding layers to map them into embeddings, which are then applied to a large language model. Fine-tuning can involve adjusting both the embedding layer and the model itself, allowing for more efficient convergence and improved output quality. This approach supports the generation of text, image, or multi-modal outputs.

Low-Rank Adaptation

Low-Rank Adaptation (LoRA) is a technique employed to efficiently fine-tune models by introducing low-rank matrices that adjust model outputs without significantly increasing parameter counts. LoRA focuses on specific model components, such as attention heads, to reduce computational demands while enhancing performance. This method is particularly beneficial for fine-tuning large models in resource-constrained environments, allowing for effective adaptation without full retraining.