Invention Title:

CONTROLLING ROBOTS USING LANGUAGE MODEL GENERATED PROGRAMS

Publication number:

US20260316268

Publication date:
Section:

Performing operations; transporting

Class:

B25J9/1658

Inventors:

Applicant:

Smart overview of the Invention

The patent application introduces a method for controlling robots using language model generated programs (LMPs). These programs are created by a code generation neural network, which is trained on computer code data. This approach allows robots to be controlled through natural language instructions, enabling them to interpret and execute tasks based on user commands. The system is designed to bridge the gap between language and physical actions, overcoming limitations of conventional methods that struggle with unseen instructions and require extensive training data.

System Implementation

The described system operates across multiple computers and locations, controlling robots as they interact with their environment. It uses pre-trained language models to generate robot policy code in response to natural language commands. These programs can autonomously generate new policy code by recomposing available API calls, allowing the system to generalize to new instructions and operate on various API sets. This flexibility enables robots to move within their environment and respond to natural language queries.

Robot Policy System

An example robot policy system is described, which includes a robot interacting with its environment to perform tasks such as navigating, identifying, and manipulating objects. The robot is equipped with sensors like cameras and microphones, which provide data for processing by machine learning models. These models help in object localization and detection, aiding the robot in executing tasks specified by natural language instructions.

Control Mechanism

The system controls the robot using control primitives, which are functions mapping parameters to control inputs. These inputs guide the robot's movements, such as adjusting velocity or relocating objects. The control system translates the primitive parameters into actionable commands for the robot, enabling precise control over its actions. This mechanism allows for seamless execution of tasks as outlined by the language model generated programs.

Neural Network Functionality

A code generation neural network is central to the system, processing input sequences in programming languages to generate output sequences that form computer programs. Trained on a corpus of computer code, it predicts subsequent tokens in a sequence, using an auto-regressive model to generate text conditioned on previous tokens and context input. This process allows the network to create programs that direct the robot's actions based on natural language commands, enhancing the robot's ability to perform complex tasks autonomously.