JP7523384B2 - CONTROLLER AND METHOD FOR CONTROLLING THE OPERATION OF A ROBOT TO PERFORM A TASKS - Patent application - Google Patents

Description

The present invention relates to robot control, and more specifically to learning and controlling robots to perform specific tasks.

Robot manipulation is central to achieving the promise of robotics. The very definition of a robot requires that it has actuators that it can use to affect the world. The potential for autonomous manipulation applications is huge: robots capable of manipulating their environment could be deployed in hospitals, elderly care and daycare, factories, outer space, restaurants, the service industry, and homes. These diverse deployment scenarios, as well as the wide range and unsystematic environmental variations in highly specialized scenarios such as food preparation, suggest that there is a need for rapid learning for robots. Many approaches focus on the problem of how a robot should learn to manipulate the world around it.

For example, some methods teach robots to learn individual manipulation skills from human demonstrations to perform new tasks. Some solutions use carefully designed demonstrations that are tailored to a specific robot. However, these methods are very laborious, time-consuming, and/or resource-intensive. Some other methods attempt to use hierarchical learning, curriculum learning, and option frameworks. These methods learn more complex tasks from a set of less complex tasks that were previously learned. This makes the goal of the new task more complex rather than performing the new task.

Some other methods use reinforcement learning (RL) to train a robot to learn how to perform new tasks in an effective manner. However, when it comes to learning new tasks for robotic manipulation, RL expects to choose from a finite set of actions, whereas many actions are effectively unbounded.

Therefore, there is a need to provide methods that simplify robot learning, thereby enabling robots to perform new tasks more efficiently.

An objective of some embodiments is to provide a system and method for learning and controlling a robot to perform a task. Additionally or alternatively, an objective of some embodiments is to modify the learning process of a robot. Some embodiments are based on the recognition that a robot that is trained to learn new skills to perform a new task can be modified to learn to reuse existing task-agnostic skills to "mimic" the new task. Such a formulation allows the embodiments to replace the learning objective of the new skill with a selection objective. Furthermore, such a formulation allows the embodiments to distance themselves from the specifics of the new task to "mimic" only the effects or effects of the new task. Additionally or alternatively, such a formulation allows some embodiments to adaptively select skills in real time to take into account possible obstacles and the nature of the new task. Additionally or alternatively, such a formulation allows some embodiments to train a robot to learn how to perform a task in an effective manner using reinforcement learning (RL).

For example, an objective of some embodiments is to have a robot learn to perform a task from multiple demonstrations of the task's execution where states and actions are observed but skills are not. Indeed, some embodiments are based on the recognition that while there are often cases where state/action pairs are observable from demonstrations, skills are unknown or undefined. Moreover, state/action pairs are more important than skills, because while state/action pairs actually perform a task in the spatial domain of the robot, skills connect sequences of state/action pairs in the time domain.

Therefore, some embodiments train a function to minimize the probability of difference between the action selected for a state and the corresponding action from multiple demonstrations of the task. The embodiments train a function to select a skill to perform an action. The learning of the function can be thought of as robot learning. The selected skill can be task agnostic as long as the skill returns the required action for the input state. Indeed, some embodiments are based on the recognition that for a new task that specifies a particular action for a particular state, there are many existing skills that happen to return the correct action for one state value, while returning the wrong action for a different state value. However, for that different state value, there may be another skill that can return the correct action.

Thus, some embodiments learn a function to reuse existing skills and replace the learning objective with a selection objective. Such selection may be performed deterministically or adaptively to take into account possible obstacles and the nature of the task. A side effect of the embodiments may include the need to execute the learned function for each control step to execute the skill, even if the selected skill is a multi-step skill configured to be executed for multiple control steps. However, the selection objective allows for simplifying the reward function for reinforcement learning.

Specifically, reinforcement learning (RL) is an area of machine learning that deals with how a software agent should take actions in an environment to maximize some notion of cumulative reward. An RL agent interacts with its environment in discrete time steps. At each time t, the agent receives an observation o _t , typically including a reward r _t . The RL controller then selects an action a _t from a set of available actions to increase the reward, _which is then sent to the environment. However, for learning new tasks for robotic manipulation, RL expects selection from a finite set of actions, while many actions are effectively unbounded. Thus, the selection-based RL of some embodiments replaces the bounded selection of actions with a bounded selection of skills from a finite library of skills. In effect, such selection-based RL simplifies the learning of task-specific functions using a library of task-agnostic skills. For example, in one embodiment, a deep reinforcement learner (DRL) determines a value for each skill and its actions.

Different embodiments use different methods to train the parameterized function that forms the RL controller. For example, in some embodiments, the parameterized function is trained using one of a deep deterministic policy gradient method, an advantage-actor critic method, a proximal policy optimization method, a deep Q-network method, or a Monte Carlo policy gradient method.

Thus, one embodiment discloses a controller for controlling the operation of a robot to perform a task. The controller includes an input interface for accepting a current state of the robot for each control step, and a memory, the memory configured to (1) store a library of skills for the robot, each skill being a probability function of the robot's state that returns a distribution of actions in response to submitting the current state, the skills being task agnostic, and the memory configured to (2) store a learned function for the task, the learned function being a probability function of the robot's state that returns a distribution of skills in response to submitting the robot's state. The controller further includes a processor, the processor configured to execute the learned function for the current state to select a skill with the highest probability on the distribution of skills for each control step in a sequence of control steps that reaches an end condition, and execute the selected skill for the current state to select an action with the highest probability on the distribution of actions to transition the robot's state from the current state to a next state. The controller further includes an output interface configured to command the robot to perform the selected action to perform the task.

Another embodiment discloses a method for controlling the movement of a robot to perform a task. The method uses a processor coupled to a memory, the memory storing (1) a library of skills for the robot, each skill being a probability function of the robot's state that returns a distribution of actions in response to submitting a current state, the skills being task agnostic, the memory storing (2) a learned function for the task, the learned function being a probability function of the robot's state that returns a distribution of skills in response to submitting a state of the robot, the processor coupled to the stored instructions, which when executed by the processor to perform steps of the method include accepting a current state of the robot for each control step, executing the learned function for the current state to select a skill that has the highest probability on the distribution of skills for each control step in a sequence of control steps that reaches an exit condition, executing the selected skill for the current state to select an action that has the highest probability on the distribution of actions to transition the state of the robot from the current state to a next state, and outputting an instruction to the robot to perform the selected action to perform the task.

DEFINITIONS In describing embodiments of the present invention, the following definitions are applicable throughout (including above).

"Robot" refers to a machine whose fundamental purpose is to perform one or more tasks automatically.

"Task" refers to a task to be performed by a robot, the task being defined by a sequence of skills in the time domain. Alternatively, a task refers to a sequence of state/action pairs to be performed in the time domain by a robot.

"Skill" refers to a probability function of the robot's state that returns an action or a distribution over actions in response to submitting the current state.

"State" refers to the values of a robot and its environment at a particular time t. These values define the configuration of the robot at a particular time t. Examples of robot configurations include, for example, motor configurations, joint configurations, gripper configurations, tool configurations, and robot position configurations. Examples of values in the environment are, for example, the positions of one or more objects, the positions of one or more obstacles, the distance to a goal position, and an image of the environment.

An "action" refers to a vector of values that transitions the robot from its current state to its next state at a particular time t. The vector of values determines the next configuration of the robot at a particular time t.

A "control step" refers to taking an action on a state at a particular time t.

"Computer" refers to any device capable of accepting structured input, processing the structured input according to prescribed rules, and producing the results of the processing as output. Examples of computers include, for example, general purpose computers, supercomputers, mainframes, superminicomputers, minicomputers, workstations, microcomputers, servers, interactive televisions, hybrid combinations of computers and interactive televisions, and special purpose hardware that emulates computers and/or software. A computer may have a single processor or may have multiple processors that may or may not operate in parallel. Furthermore, a computer refers to two or more computers connected via a network for transmitting and receiving information between the two or more computers. Examples of such computers include distributed computer systems for processing information via computers linked by a network.

"Central processing unit" or "processor" refers to a computer or a component of a computer that reads and executes software instructions.

"Memory" or "computer-readable medium" refers to any storage accessible by a computer for storing data. Examples include magnetic hard disks, floppy disks, optical disks such as CD-ROMs or DVDs, magnetic tape, memory chips, carrier waves used to carry computer-readable electronic data such as those used to send and receive email or access a network, and computer memory such as random access memory (RAM).

"Software" refers to the prescribed rules for operating a computer. Examples of software include code segments, instructions, computer programs, and programmed logic. Software in an intelligent system may be self-learning.

A "module" or "unit" refers to a basic component in a computer that performs a task or part of a task. A "module" or "unit" may be realized in either software or hardware.

"Controller" and/or "control system" refers to a device or set of devices for managing, commanding, directing, or coordinating the behavior of other devices or systems. A controller may be realized by hardware, a processor whose operations are configured by software, or a combination of both. A controller may be an embedded system.

The presently disclosed embodiments are further described with reference to the accompanying drawings. The drawings shown are not necessarily to scale, emphasis instead generally being placed on illustrating the principles of the presently disclosed embodiments.

FIG. 1 shows a schematic overview of the principles used by some embodiments to learn and control the behavior of a robot to perform a task. FIG. 1 illustrates a block diagram of a robot control system for learning and controlling the behavior of a robot to perform a task. FIG. 1 illustrates an example skill stored in a skill library module of a robotic control system. FIG. 1 illustrates a task demonstration for a robot using a remote control device, according to some embodiments. FIG. 1 illustrates a task demonstration for a manually operated robot, according to some embodiments. FIG. 2 illustrates an example task demonstration stored in a task demonstration module, according to some embodiments. FIG. 2 illustrates an example architecture diagram of a learning module that learns a function for selecting a skill, according to some embodiments. FIG. 1 illustrates an example architecture for learning a function using selection-based reinforcement learning, according to some embodiments. FIG. 2 illustrates an example architecture diagram of an execution module for executing tasks, according to some embodiments. FIG. 4 illustrates a flowchart illustrating operations performed by an executive module according to some embodiments. FIG. 2 illustrates an exemplary learning and execution phase for performing a task, according to some embodiments.

In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that the present disclosure may be practiced without these specific details. In other instances, the devices and methods are shown only in block diagram form in order to avoid obscuring the present disclosure.

As used herein and in the claims, the terms "for example," "for example," and "such as," as well as verbs such as "comprise," "have," "include," and other verb forms thereof, when used in connection with a list of one or more components or other items, should each be construed as open-ended, meaning that the list should not be considered to exclude other additional components or items. The term "based on" means based at least in part on. Furthermore, it should be understood that the phraseology and terminology used herein are for descriptive purposes and should not be considered limiting. Any headings utilized in this description are for convenience only and have no legal or limiting effect.

FIG. 1 shows a schematic overview of the principles used by some embodiments to learn and control the behavior of a robot to perform a task, according to some embodiments. A robot is a machine whose fundamental purpose is to perform one or more tasks automatically. A task is a task that is difficult or repetitive for a human to perform. For example, tasks correspond to industrial tasks, household maintenance, and serving in a restaurant. In some embodiments, an untrained robot 102 refers to a robot that has not been trained or is learning. To make a robot (e.g., an untrained robot 102) perform a task, the robot needs to be extensively trained on the task. However, the task is complex and involves several steps to be performed. Therefore, to help the robot learn, the robot's task can be divided into subtasks (hereinafter referred to as skills). The task can be to assemble several parts into some complex object. Skills required for assembly may include moving the robot arm to the left, opening the gripper, lowering the gripper, lifting the part, raising the arm, moving the arm to the right to a specific position, lowering the arm, inserting the part within another part, etc.

Some embodiments are based on the recognition that a robot that has been trained for a task can be operated to perform a new task. For example, a robot that has been trained to maintain a household, such as washing dishes and sweeping the floor, can be operated to wash dishes, sweep the floor, and dispose of trash, etc. However, in order to operate the robot, the robot still needs to learn the new task quickly and extensively. Therefore, the objective of some embodiments is to train the robot to learn a new task. As shown in FIG. 1, an untrained robot 102 is trained by a robot control system 104, thereby forming a trained robot that performs the new task. In some embodiments, the robot control system 104 includes a learning module 104a that trains the untrained robot 102 to learn individual operation skills from human demonstrations. For example, the learning module 104a uses carefully designed demonstrations that are adapted to the specifics of the robot. However, this approach requires additional work to design the demonstrations, which is a long and time-consuming process. In some other embodiments, the learning module 104a uses reinforcement learning (RL) to train the robot to learn new tasks in an efficient manner. In reinforcement learning (RL), the RL agent interacts with its environment at discrete time steps. At each time t, the agent receives an observation o _t , which typically includes a reward r _t . The RL controller then selects an action a _t from a set of available actions to increase the reward, and the action a _t is then sent to the environment. However, for learning new tasks for robotic manipulation, RL expects to select from a finite set of actions, while many actions are effectively unbounded. Furthermore, RL requires the need to design a reward function appropriate for learning new tasks. Designing these reward functions is difficult.

Therefore, an objective of some embodiments is to provide an improved learning method to perform new tasks. Thus, the robot control system 104, including the learning module 104a with learning objectives, is modified by the learning module 108a with selection objectives. The learning module 108a trains the unskilled robot 102 to reuse existing task-agnostic skills to transform the unskilled robot into a learned robot 106. Furthermore, the learning module 108a allows the embodiment to distance itself from the specifics of the new task to "mimic" only the effects or effects of the new task. Additionally or alternatively, the learning module 108a allows some embodiments to adaptively select skills in real time to take into account possible obstacles and the nature of the new task. Furthermore, in some embodiments, the selection objective allows the reward function for reinforcement learning to be simplified. In some embodiments, the learning module 108a uses RL to train the unskilled robot 102 to perform the new task in an efficient manner. Specifically, the learning module 108a uses RL to train the unskilled robot 102 to select skills from a finite library of existing task-agnostic skills. Thus, the selection-based RL of some embodiments replaces the bounded selection of actions with a bounded selection of skills from a finite library of skills.

FIG. 2 shows a block diagram of a robot control system 200 for learning and controlling the movement of a robot to perform a task, according to some embodiments. Hereinafter, "robot control system" and "controller" may be used interchangeably and mean the same thing. The controller 200 includes an input interface and an output interface for connecting the controller 200 to other systems and devices. In some embodiments, the input interface includes a human machine interface (HMI) 202 and an imaging interface 220. The HMI 202 may connect the controller 200 to an input device such as a keyboard 230 and/or a pointing device/medium 232. The pointing device 232 may include, for example, a mouse, a trackball, a touchpad, a joystick, a pointing stick, a stylus, or a touch screen. The imaging interface 220 may be adapted to connect the controller 200 to an imaging device 226. The imaging device 226 may include an RGBD camera, a depth camera, an RGB camera, a grayscale camera, a computer, a scanner, a mobile device, a webcam, or any combination thereof. According to some embodiments, the input interface may be configured to receive/accept a current state of the robot 236 for each control step of the robot 236. The input interface further includes a network interface controller (NIC) 212 adapted to connect the controller 200 to the network 234 via the bus 210. The controller 200 receives the current state of the robot 236 for each control step via the network 234, either wirelessly or wired.

According to some embodiments, the input interface may further be configured to receive a set of executions of the task. The set of executions is a demonstration of the task for the robot to execute the task. Hereinafter, "execution set of the task" and "task demonstration" may be used interchangeably and mean the same thing. Some embodiments are based on the recognition that the imaging interface 220 may receive a task demonstration for the robot 236 from the imaging device 226. In some embodiments, the imaging device 226 may provide the task demonstration in image and/or video format. Additionally or alternatively, the controller 200 may include an image processing module 218. The image processing module 218 is configured to process images and videos to provide the task demonstration as an input to the learning module 206d or to store the task demonstration in the task demonstration module 206b. In some embodiments, the image processing 218 may augment the task demonstration input to the learning module 206d or augment the task demonstration stored in the module 206b. Additionally or alternatively, the controller 200 includes a task processing module 216 that may be configured to receive task demonstrations from the robot 236 directly or via the network 234. In some embodiments, the task processing module 216 may be further configured to process the task demonstrations to provide them as input to the learning module 206d or to store them in the task demonstration module 206b. The task processing module 216 takes the task demonstrations in some format and converts them into a sequence of state/action pairs. For example, the tasks may be received as robot commands and the task processing module 216 may convert those commands into state/action pairs. The task processing module 216 may be used in augmenting the task demonstrations from elements provided by the image processing 218.

The controller 200 further includes a processor 204 and a memory 208 that stores instructions executable by the processor 204. The processor 204 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 208 may include random access memory (RAM), read only memory (ROM), flash memory, or any other suitable memory system. The processor 204 is connected to one or more input and output devices via a bus 210. The stored instructions implement a method for learning and controlling the behavior of a robot (e.g., robot 236) to perform a task.

In some embodiments, the memory 208 may be further expanded to include a storage 206. In some embodiments, the storage 206 includes a learned skills library module 206a, a task demonstration module 206b, a robot controller 206c, a learning module 206d, and an execution module 206e. The skills library module 206a may be configured to store a library of skills (i.e., a plurality of skills). A skill is a probability function of a state of a robot (e.g., robot 236) that returns an action or a distribution over actions in response to submitting the current state of the robot. Some embodiments are based on the recognition that the skills in module 206a are subsequences or subtasks of a task. Furthermore, each skill may be a state/action pair or a sequence of state/action pairs in the time domain. As such, the skills in module 206a may be single-step or multi-step skills. For example, skills corresponding to state/action pairs may be coined as single-step skills, and skills corresponding to sequences of state/action pairs may be coined as multi-step skills. Further, the skills stored in module 206a are described in more detail with reference to FIG. 3.

Some embodiments are based on the recognition that the task demonstration module 206b may be configured to store a set of executions of a task (i.e., a task demonstration for the robot 236). In some embodiments, the task demonstration is observed or recorded by an input device as a sequence of state/action pairs, while the skills are unknown or undefined. Therefore, the task demonstration is stored in the module 206b as a sequence of state/action pairs. Furthermore, the state/action pairs are more important than the skills, because the state/action pairs actually perform the task in the spatial domain of the robot, while the skills connect the sequence of state/action pairs in the time domain. Task demonstrations for robots are described in more detail with reference to Figures 4A-4C.

The robot controller 206c may be configured to store instructions that, when executed by the processor 204, execute one or more modules in the storage 206. Some embodiments are based on the recognition that the robot controller 206c manages the learning and execution phases of the robot.

The learning module 206d may be configured to store instructions that, when executed by the processor 204, cause the processor 204 to learn a function for selecting skills. Some embodiments are based on the recognition that the learning module 206d stores instructions for receiving a set of executions of a task, each of which defines a sequence of state/action pairs, instructions for determining, for each state, a distribution of actions from the sequence of state/action pairs that fit the action received for that state, and instructions for determining, for a state, a skill that has the highest probability of returning an action sampled on the distribution of actions determined for that state. The objective of some embodiments is to learn a function to select skills that penalize switching between skills. To that end, some embodiments use reinforcement learning (RL) to learn the function. For example, RL includes a reward function that includes a penalty for switching between skills and a penalty based on the difference between the action selected by RL and the action of the demonstrated task. Further, the learning module 206d (i.e., the learning phase) for learning a robot to perform a task is described in detail with reference to FIGS. 5-6.

The execution module 206e may be configured to store a learned function for selecting a skill. The learned function is a single argument function. In some embodiments, the learned function is a probability function of the robot's state that returns a distribution of skills in response to submitting the robot's state. Furthermore, the execution module 206e may be configured to store instructions for selecting an action from the selected skill. Furthermore, the execution module 206e is configured to store instructions for performing a skill selection for each control step, the skill corresponding to a multi-step skill. Further, the execution module 206e for executing a task (i.e., the execution phase) is described in detail with reference to FIGS. 7-9.

The output interface may be configured to instruct the robot to perform a selected action to perform the task. The output interface may include a display interface 214 and an application interface 222. The display interface 214 may connect the controller 200 to a display device 224. The display device 224 may include, for example, a computer monitor, a camera, a television, a projector, or a mobile device. The application interface 222 may connect the controller 200 to an application device 228. In some embodiments, the application device may be embodied within the controller 200. As such, the application device 228 operates based on the selected action to perform the task. For example, the application device 228 provides a control signal for the robot to perform the selected action to perform the task.

3 illustrates an example skill stored in the skill library module 206a according to some embodiments. Some embodiments are based on the recognition that the skills stored in the skill library module 206a are basic skills such as move option, rotate option, stop option, pick option, and place option. Thus, the skill library module 206a includes skills such as move left 302, move right 304, move in XYZ directions 306, move up 308, move down 310, stop 312, place object 314, pick object 316, and rotate tool 318. In some embodiments, the skills stored in the skill library module 206a may be composed of two or more basic skills that lead to a complex skill. For example, a complex skill may include basic skills such as move in XYZ directions 306 and place object 316. Some embodiments are based on the recognition that there are multiple libraries of skills in the module 206a to perform a task or tasks. For example, if a task requires skills from multiple libraries, module 206a stores multiple libraries of skills. In some other embodiments, there may be a need to learn new skills to perform some task demonstrations. Therefore, the learned new skills may be stored in the library of existing skills, or a new library of skills may be created to store the learned new skills. The decision to learn a new skill is entirely dependent on the decision of a human expert. The objective of some embodiments is to maintain all the necessary skills in the skill library module 206a to perform a task.

Furthermore, each skill is stored in module 206a as a probability function. Thus, a skill in module 206a is a probability function of the robot's state that returns an action or a distribution over actions in response to submitting the current state. Thus, mathematically, it is defined as follows:

Therefore, the skills in module 206a are:

It can be expressed mathematically as follows:

4A illustrates a task demonstration for a robot using a remote control device, according to some embodiments. As shown in FIG. 4A, the remote control device 402 can be used to demonstrate a task for the robot 236. Some embodiments are based on the recognition that an operator 404 (e.g., a technician, a user, etc.) operates the remote control device 402, which then operates the robot 236 to demonstrate a specific task. In some embodiments, the remote control device 402 transmits robot configuration settings (i.e., robot settings) to the robot 236 to demonstrate a specific task. For example, the remote control device 402 transmits control commands such as movement in XYZ directions, velocity control commands, and joint position commands, etc., to demonstrate a specific task. For example, as shown in FIG. 4A, a specific task can be defined by two skills, such as a skill to pick an object and a skill to move the object upwards. The remote control device 402 transmits the robot configuration settings for both skills to demonstrate a particular task. In some other embodiments, the demonstration of a particular task occurs in a control step. For example, the remote control device 402 transmits the robot configuration settings for each control step. Thus, the particular task to be demonstrated for the robot 236 can be recorded in state/action pairs by the input device. The input device can provide the recorded data to the controller 200.

FIG. 4B illustrates a task demonstration for a manually operated robot, according to some embodiments. As shown in FIG. 4B, the robot 236 can be manually operated to demonstrate a specific task. Some embodiments are based on the recognition that to demonstrate a specific task, the operator 404 manually controls the settings of the robot 236, such as the motors, joints, grippers, tools, and position of the robot 236. In some embodiments, a task demonstrated by manual operation may not define skills and skill configurations for the demonstrated task. Therefore, the task demonstration is recorded by an input device in state/action pairs in the time domain. The input device may provide the recorded data to the controller 200.

In some other embodiments, the task demonstration may be performed by simulation. In some other embodiments, the task demonstration may be the execution of a wrist point in 3D space that is reached during the task demonstration, for example, for the movement of an end of the robot, such as a tool. In some other embodiments, the task demonstration may be achieved by a combination of manual and remote control device operations. In some other embodiments, the imaging device 226 may record the task demonstration, and the image processing 218 may process the recorded task demonstration to augment the task demonstration with visual elements. Furthermore, the task demonstration for the robot may not be limited to the exemplary description above. Instead, all types of demonstrations that are within the scope of the present invention should be considered.

4C illustrates an exemplary task demonstration stored in the task demonstration module 206b according to some embodiments. The purpose of some embodiments is to store a set of executions of a task (i.e., a task demonstration for a robot) received from an input device. Some embodiments are based on the recognition that a task demonstration is received in state/action pairs, while skills are unknown or undefined. Furthermore, state/action pairs are more important than skills, because state/action pairs actually perform a task in the spatial domain of the robot, while skills connect a sequence of state/action pairs in the time domain. Thus, a received task demonstration may be stored in the task demonstration module 206b as a sequence of state/action pairs. In some embodiments, the sequence of state/action pairs is defined in the time domain. For example, a task demonstration is represented as a sequence of state/action pairs from a start time to an end time.

A state S _t at a particular time t includes values that define the current configuration of the robot and may be augmented by elements from its environment. In some embodiments, the configuration of the robot includes, but is not limited to, settings of motors, joints, grippers, tools, positions, images, or any combination thereof. Values represent, for example, joint values for a particular robot configuration. Examples of values in the environment are the positions of one or more objects, the positions of one or more obstacles, the distance to a goal position, an image of the environment. An action A _t at a particular time t includes a vector of values to determine different configurations of the robot. Some embodiments are based on the realization that an action moves the robot from one state to the next. The action may be discrete or continuous in nature. An action may be a single value or a vector of values, etc.

In FIG. 4C, block 406 may represent a task demonstration stored in task demonstration module 206b. Additionally, task demonstration module 206b stores several task demonstrations, not just task demonstration 406. For example, task demonstration module 206b stores a finite number of task demonstrations. Block 460 may represent the last or final task demonstration stored for the robot. Task demonstration 406 starts at time _t0 and stores states and actions 406-0 (e.g., state/action pair S _t0, A _t0 ) at time _t0 . In some embodiments, an action (i.e., A _t0 ) to be taken for a state (i.e., S _t0 ) is defined as a control step for time _t0 . Task demonstration module 206b continues to store task demonstration 406 until the end time of task demonstration 406. At the end time _tN , the task demonstration module 206b stores the state and action 406-N (e.g., state/action pair S _tN , A _tN ). Thus, the task demonstration 406 includes N+1 state/action pairs for N+1 time instances. At time _tN , when the final action A _tN is demonstrated for the task demonstration 406, the robot is in state S _tN+1 . Thus, the state 406-N+1 is further stored as the task demonstration 406. According to some embodiments, the stored task demonstration 406 includes a trajectory

can be expressed as:

Additionally, the task demonstration module 206b stores all finite number of task demonstrations as described with reference to the task demonstration 406. According to some embodiments, the task demonstrations in the task demonstration module 206b may have the same or different sequences of state/action pairs. For example, the state/action pairs 460-0 to 460-O may be the same or different from the state/action pairs 406-0' to 406-N. Furthermore, the first state and action pair for a demonstration may be the same or different. Furthermore, the task demonstrations in the task demonstration module 206b may have the same length or different lengths. For example, the task demonstration 460 may include O+1 state/action pairs for O+1 time instances, and the task demonstration 406 may include N+1 state/action pairs for N+1 time instances. Thus, the task demonstration module 206b stores M finite task demonstrations, where M≧1. According to some embodiments, the M finite demonstration tasks are a set of trajectories D, i.e.,

can be expressed as:

5 shows an example architecture diagram of the learning module 206d for learning a function for selecting a skill, according to some embodiments. The learning module 206d may be configured to learn a function for selecting a skill from the skill library module 206a based on the task demonstration in the module 206b. Learning the function may be considered as training the robot to select a skill. In some embodiments, the skills in the skill library module 206a are task agnostic to perform a task. Some embodiments are based on the recognition that the task demonstration in the task demonstration module 206b, which specifies a specific action for a specific state, does not have an indication of which skill should be selected from the skill library module 206a to perform the task. Thus, the objective of some embodiments is to learn a function for selecting a skill to perform a task.

Some embodiments are based on the recognition that the learning module 206d is configured to receive a set of task executions (i.e., task demonstrations for the robot) from at least one of the task processing module 216, the image processing module 218, or the task demonstration module 206b. The received task demonstrations are sequences of state/action pairs, as exemplarily shown in FIG. 4C. The received demonstrations are demonstrations of the tasks that need to be executed by the robot at runtime. In some embodiments, all task demonstrations (i.e., all of the M finite task demonstrations for a particular task) may be received to train the function. In some embodiments, the received M finite task demonstrations are a set of trajectories D, i.e.,

The formula is expressed as:

is equivalent to

Some embodiments are based on the recognition that the learning module 206d is further configured to determine, for each state in the set of trajectories D, a distribution of actions that fit the actions received for that state. For illustrative purposes, consider state S _t0 . At time _t0 , state S _t0 receives M finite A _t0 actions from M finite task demonstrations in set D. In some embodiments, the M finite A _t0 actions from the M finite task demonstrations may be the same or different. Thus, the learning module 206d determines, for state S _t0 , a distribution over M finite A _t0 actions. Further, the learning module 206d iterates to determine the distribution of actions for each state in the set of trajectories D as described with respect to state S _t0 . Thus, the learning module 206d determines the distribution of actions for all states in set D. Further, the learning module 206d for determining, for each state in the set of trajectories D, a distribution of actions that fit the actions received for that state is described in detail with reference to FIG. 9.

In some embodiments, the learning module 206d is configured to sample, for each state in the set of trajectories D, an action on the distribution of actions determined for the corresponding state. Some embodiments are based on the realization that the sampled action corresponds to the most probable action on the distribution of actions. Indeed, the objective of some embodiments is to determine, for a state in the set of trajectories D, a skill for the module 206a based on the action sampled for that state. Therefore, a function needs to be trained to determine, for a state, a skill from the module 206a that corresponds to the most probable return of the action sampled on the distribution of actions determined for that state.

In some embodiments, learning module 206d is configured to learn a function for determining the skill from module 206a. Learning the function is described in further detail below.

For clarity of explanation, we consider a single demonstration of the task, i.e., M=1. The problem is formulated as follows:

Information available:
A sequence of actions is represented as A=A _t0:tN .

The sequence of states is represented as S=S _t0:tN+1 .

The set of discrete skills in module 206a is represented as H={H ₁ , H ₂ , . . . , H _j }.

Information to be derived:
Determining the sequence of skills to be selected that maximizes the likelihood is denoted as L _θ (H, S, A). L _θ (H, S, A) represents that the discrete skills selected in the sequence from the set of skills H make the observations S, A most likely, i.e., have the highest probability. In some embodiments, the discrete skills selected in the sequence are a subset of or equal to the set of skills H. Thus, maximizing the sequence of skills H can be achieved from a maximum likelihood optimization. Furthermore, using Bayesian probability theory and eliminating terms that do not play a role in the optimization, maximizing L _θ (H, S, A) can be expressed as:

is equivalent to

Since H is unknown in L _θ (H, S, A),

and ignore it as follows:

π ^h (A|S) in equation (1) is a previously learned skill from the skill library module 206a. f _θ (S) in equation (1) is a function that returns a skill distribution given a state. For example, the function f _θ (S) is a state probability distribution function that returns a distribution over skills given an input state S. According to some embodiments, f _θ (S) provides the ability to formulate a termination function. The termination function indicates whether the current skill is terminated (represented by a value of 1) or the current skill continues (represented by a value of 0) given a state S. For example, the termination function is defined as the function β(S)→0/1.

Some embodiments are based on the realization that the action or distribution of actions given by π ^h (A|S) for state S _t should match as closely as possible to the distribution of actions determined for state S _t (i.e., the distribution of actions determined for state S _t from M finite demonstrations) to determine the skill π ^h (A|S) for state S _t . To that end, the learning module 206 d generates constraints on equation (1), e.g., the constraints being

The action A _t in the constraint may be a sampled action determined for state S _t . In some embodiments, the action A _t may be one of the actions on the distribution of actions determined for state S _t .

may be the minimum difference value (e.g., a probability value or a metric such as distance) between the action returned by the skill (π ^h (A|S)) for state S _t and the distribution of actions determined for state S _t . Thus, equation (1) is transformed as follows:

Equation (2) is a constraint

is a single argument function with t = t ...

Equation (2) may be trained in a variety of different ways. In one embodiment, equation (2) may be trained by the well-known technique of expectation-maximization (EM) or some variation of EM. In another embodiment, equation (2) may be trained by a reinforcement learning (RL) technique having a reward function that minimizes the deviation of the outcome of the execution of the task. Further, a function trained based on the RL technique is described in detail with reference to FIG. 6 . Thus, a function is trained (trained function 502) to determine, for a state S _t , a skill from module 206a that corresponds to the highest probability of returning an action sampled on the distribution of actions determined for said state S _t . For example, in equation (2),

determines for state S t the skill π ^h (A|S) that has the highest probability of returning an action sampled over the distribution of actions determined for state _{S t .}

Furthermore, constraints

can be given as a soft constraint as follows:

In equation (3), p represents a norm, e.g., 0-norm or 2-norm, typically n=1, n=2. Here, p and n are so-called hyperparameters, i.e., parameters selected by a human who designs the final optimization function. In some embodiments, equation (3) may be the output of the learning module 206d as the learned function 502. In some other embodiments, the output of the learning module 206d may be a sequence of skills determined by the learned function 502 for all states in the set D. For example, a set H may be generated to store the determined skills in a sequence, i.e., H={H ₄ , H ₇ ,..., H ₁ }. Furthermore, the skills determined in the sequence may be stored in a list L. For example, a list L may be generated to store actions or distributions of actions in a sequence, i.e., L={π ₄ (s), π ₇ (s),...,π ₁ (s)}. Further, in some embodiments, the learning module 206d returns a termination function β(S). Thus, whenever the termination function β(S) returns 1, a probability function π ^{h l} associated with the next skill from the list L is selected and then executed. In some embodiments, the termination function β(S) determines the time to switch from the currently selected skill in the list L to the next skill to perform the task.

FIG. 6 shows an example architecture for learning a function using selection-based reinforcement learning, according to some embodiments. Reinforcement learning (RL) is an area of machine learning that concerns how a software agent should take actions in an environment to maximize some notion of cumulative reward. The RL agent interacts with its environment at discrete time steps. At each time t, the agent receives an observation o _t , which typically includes a reward r _t . The RL controller then selects an action a _t from a set of available actions to increase the reward, _which is then sent to the environment. However, for learning new tasks for robotic manipulation, RL expects selection from a finite set of actions, although many actions are effectively unbounded. Therefore, a selection-based RL architecture that replaces the bounded selection of actions with a bounded selection of skills from a finite library of skills is shown in FIG. 6.

In FIG. 6, block 602 may represent the task demonstration module 206b. In block 604, a state observer records a state S _t at time t in module 206b. In block 606, an action observer records an action received for state S _t from a plurality of sets of executions (i.e., M finite task demonstrations). In block 608, a reward function for state S _t is defined that minimizes the deviation of the outcome of the execution of the task. Some embodiments are based on the recognition that the reward function includes a penalty for switching skills. As such, the reward function includes a termination function β(S) for switching skills. For example, the termination function identifies a time to switch a current skill to another skill in module 206b. In other embodiments, the reward function includes a penalty based on the difference between the recorded action in module 206b and the generated action 614.

In block 610, the RL controller samples an action from the actions received for state S _t . The sampled action corresponds to the action with the highest probability for state S _t . Furthermore, the RL controller selects the skill from module 206a that corresponds to the highest probability for state S _t to return the sampled action or the action received for state S _t . So, in block 612, the RL controller selects a skill from module 206a that includes a bounded/finite number of skills. The skill returns an action or a bounded/finite distribution of actions for a given state S _t . In block 614, the action with the highest probability on the finite distribution of actions is selected. So, the selection-based RL of some embodiments replaces the bounded selection of actions with a bounded selection of skills from a finite library of skills. In effect, such a selection-based RL simplifies the learning of task-specific functions using a library of task-agnostic skills. For example, in one embodiment, a deep reinforcement learner (DRL) determines a value for each skill and its action.

Different embodiments may use different methods for learning the parameterized function that forms the RL controller. For example, in some embodiments, the parameterized function is learned using one of a deep deterministic policy gradient method, an advantage-actor critic method, a proximal policy optimization method, a deep Q-network method, or a Monte Carlo policy gradient method.

7 shows an example architecture diagram of the execution module 206e for executing a task, according to some embodiments. As shown in FIG. 7, in block 702, the execution module 206e is configured to accept the current state of the robot 236 for each control step via the input interface. In block 704, the execution module 206e is configured to store a learned function for executing the task. As shown in FIG. 5, the learned function is a distribution of skills in response to submitting the state of the robot. In block 706, the execution module 206e is configured to execute the learned function of the current state for each control step in the sequence of control steps that reaches an end condition to select the skill that has the highest probability on the distribution of skills. As shown in FIG. 3, the skill is a probability function of the state of the robot that returns an action or a distribution over actions in response to submitting the current state. In block 708, the execution module 206e is configured to execute the selected skill for the current state to select the action that has the highest probability in the distribution of actions and transition the state of the robot from the current state to the next state. In block 710, the execution module 206e is configured to output the selected robot action via the output interface to instruct the robot to perform the task.

8 is a flow chart illustrating operations performed by the execution module 206e according to some embodiments. In block 802, the execution module 206e is configured to obtain a state of the robot for a control step. In some embodiments, the state of the robot is obtained via an input interface. In block 804, the execution module 206e is configured to execute a learned function for the obtained state to select a skill that has the highest probability on the distribution of skills. Alternatively, the execution module 206e executes a learned function for the obtained state to select a skill that corresponds to the highest probability in the list L. The selected skill may be a single-step skill or a multi-step skill. In some embodiments, a multi-step skill is a skill that may be configured to execute for multiple states of multiple control steps and return their corresponding multiple actions. In block 806, the execution module 206e is configured to execute the selected skill for the obtained state to select an action that has the highest probability on the distribution of actions. In block 808, the execution module 206e is configured to output the selected action to instruct the robot to execute the selected action.

As should be understood, once the robot executes the selected action, the robot transitions from the acquired state to the next state. Additionally, the execution module 206e acquires the next state for the next control step in block 802. In some embodiments, the execution module 206e executes a termination function β(S) for the next state in block 804, and then executes the learned function for the next state. The termination function β(S) returns 0 for the next state, indicating that the skill should be continued, or 1 for selecting a skill (i.e., the termination condition). In some embodiments, the termination function β(S) returns 0 if the previously selected skill (i.e., the skill selected for the acquired state) is a multi-step skill and corresponds to returning the sampled action with the highest probability for the next state, otherwise the termination function β(S) returns 1. If the termination function returns 0, the execution module 206e skips execution of the learned function for the next state in block 804. If the termination function returns 1, the execution module 206a executes the learned function for the next state in block 804. In some other embodiments, the execution module 206e executes the learned function for the next state without executing the termination condition β(S) in block 804. Thus, the execution module 206e repeats executing the learned function for the next state even if the previously selected skill (i.e., the skill selected for the acquired state) is a multi-step skill and corresponds to returning the sampled action with the highest probability for the next state. The execution module 206e executes the selected skill in block 806 to select the action with the highest probability on the distribution of actions. The execution module 206e outputs the selected action in block 808. The flowchart 800 is further repeated until all control steps of the task are executed.

FIG. 9 illustrates an exemplary learning and execution phase for performing a task, according to some embodiments. Part 902 is a preparation phase for performing a task. As illustrated in FIG. 9, preparation phase 902 includes a skill library module 206a and a task demonstration module 206b. Module 206a stores a bounded/finite number of skills in the form of a library or libraries. For example, module 206a stores skill 906a (i.e., skill H1), skill 908a (i.e., skill H2), and skill 910a (i.e., skill H3). Each skill in the module is a probability function of the state of the robot that returns an action or a distribution of actions in response to submitting the current state. For example, skill 906a returns action _A0 for state _S0 , skill 908a returns action _A1 for state _S1 , and skill 910a returns a distribution of actions for states _S2 and _S3 . However, skill 910a returns a distribution of actions where _A2 is the most probable for state _S2 and _A3 is the most probable for state _S3 . Some embodiments are based on the recognition that the skills stored in module 206a are task agnostic.

The task demonstration module 206b stores demonstrations (i.e., multiple executions) of a task (i.e., {S ₀ -A ₀ , S ₁ -A ₁ , S ₂ -A ₂ , S ₃ -A ₃ , S ₄ }) in state/action pairs while the skills are unknown or undefined. For example, task demonstration module 206b stores task demonstration 906b (i.e., { _S0 - _A0 , _S1 - _A1 , _S2 - _A2 , _S3 - _A3 , _S4 }), task demonstration 908b (i.e., { _S0 - _A0 , _S1 - _A1 , _S2 - _A2 , _S3 - _A3 , _S4 }), and task demonstration 910b (i.e., { _S0 - _A1 , _S2 - _A2 , _S3 - _A3 , _S4 }). State _S0 is the configuration of the robot at time _t0 , and action _A0 represents a vector of values at time _t0 that transitions the robot from state _S0 to the next state _S1 . 9, task demonstrations 906b and 908b are recorded correctly. However, due to some random error, state _S1 is not recorded in task demonstration 910b.

Some embodiments are based on the recognition that the task demonstrations 906b, 908b, and 910b are sequences of state/action pairs, but do not have instructions regarding skills. Therefore, the learning module 206d trains a function to select skills from the module 206a to perform a task based on the task demonstrations 906b, 908b, and 910b in the module 206b.

Part 904 represents a learning phase of executing a task. As shown in FIG. 9, the learning module 206d is configured to receive task demonstrations in block 906d. For example, the learning module 206d receives three demonstrations 906b, 908b, and 910b. The learning module 206d is configured to determine, for each state, from the received task demonstrations (i.e., the three demonstrations 906b, 908b, and 910b), a distribution of actions that fit the actions received for that state in block 908d. Block 908d-0 represents the distribution of actions determined for state _S0 , block 908d-1 represents the distribution of actions determined for state _S1 , block 908d-2 represents the distribution of actions determined for state _S2 , and block 908d-3 represents the distribution of actions determined for state _S3 . For example, state S ₀ receives two A ₀ actions from task demonstrations 906b and 908b, and action A ₁ from task demonstration 910b. Thus, a distribution 908d-0 is determined for state S ₀ that fits actions A ₀ and A _1. Further, distribution 908d-0 returns action A ₀ with a probability of 2/3 and action A ₁ with a probability of 1/3. Similarly, distributions 908d-1, 908d-2, and 908d-3 are determined. In some embodiments, the learning module 206d is configured in block 908d to sample an action for each state in the task demonstration on the distribution of actions determined for the corresponding state. In some embodiments, the sampled action corresponds to the action with the highest probability on the distribution of actions. Thus, the sampled actions for states S ₀ , S ₁ , S ₂ , and S ₃ are A ₀ , A ₁ , A ₂ , and A ₃ , respectively.

In some embodiments, the state may differ between demonstrations 906b, 908b, and 910b. The indexes and notations used here are for clarity of explanation. The same arguments apply to the actions. For example, if the states differ, additional processes outside the scope of the present invention may be invoked to determine alignment according to the most similar state among task demonstrations 906b, 908b, and 910b. An example of such alignment may be the well-known process of dynamic time warping.

The learning module 206d is configured to determine, in block 910d, for a state, the skill that has the highest probability of returning a sampled action on the distribution of actions determined for that state. To this end, the learning module 206d performs the function

The learned function

For state S _t , determine the skill that has the highest probability of returning an action sampled on the distribution of actions determined for state S _t . For example, the learning module 206d determines skills H1 and H2 for states S ₀ and S ₁ , respectively, and determines skill H3 for states S ₂ and S _3. Skill H3 is a multi-step skill. Alternatively, the skills determined in the learning phase may be stored as a sequence of skills in a set H. For example, set H is H={H ₁ , H ₂ , H ₃ }, which, when executed in sequence by the robot as in set H, completes the task in the execution phase.

As should be appreciated, once the learning phase is complete, flow continues with the execution phase, whereby an execution module 206e is provided to execute the tasks.

The execution module 206e is configured to receive a current state. For example, the execution module 206e receives a state _S0 . Then, the execution module 206e is configured to execute the learned function for the state _S0 to select a skill with the highest probability on the distribution of skills. For example, the execution module 206e selects a skill H1 for the state _S0 because the skill H1 for the state _S0 is the skill determined in the learning phase, and returns an action _A0 with the highest probability. Further, the execution module 206e is configured to execute the selected skill H1 for the state _S0 to select the action _A0 . The execution module 206e instructs the robot to execute the action _A0 for the state _S0 .

As should be understood, when the robot executes the action _A0 , the robot transitions to state _S1 . The execution module 206e again receives state _S1 and executes the termination function β(S) for state _S1 to determine whether to select another skill for state _S1 . Skill H1 provides an action or a distribution of actions only for state _S0 . Thus, the termination function returns 1 for state _S1 . When the termination function returns 1, the execution module 206e is configured to select another skill for state _S1 as described with respect to state _S0 . This process is repeated until the execution module receives state _S4 . Thus, the execution module 206e selects skill H2 for state _S1 and skill H3 for states _S2 and _S3 . Skill H3 is a multi-step skill executed by the execution module 206e for states _S2 and _S3 to return actions _A2 and _A3 , respectively. Once action _A3 is executed, the robot transitions to state _S4 , which indicates completion of task execution, whereby the skill to be executed by the robot to perform the task is selected from a library of task-agnostic skills, thereby replacing the learning objective with the selected objective.

The above description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments will provide one of ordinary skill in the art with an enabling description for implementing one or more exemplary embodiments. Various changes are contemplated that may be made in the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter as set forth in the appended claims.

Specific details are given in the following description to provide a thorough understanding of the embodiments. However, as will be understood by those skilled in the art, the embodiments may be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other cases, well-known processes, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments. Additionally, like reference numbers and names in the various drawings indicate like elements.

Furthermore, particular embodiments may be described as a process that is depicted as a flowchart, flow diagram, data flow diagram, structure diagram, or block diagram. Although a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Furthermore, the order of operations may be rearranged. A process may be terminated when its operations are completed, but may have additional steps not discussed or included in the diagram. Furthermore, not all operations in any particularly described process may be performed in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, or the like. When a process corresponds to a function, the end of the function may correspond to a return of the function to the calling function or to the main function.

Furthermore, embodiments of the disclosed subject matter may be implemented, at least in part, either manually or automatically. The manual or automatic implementation may be performed or at least assisted through the use of machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments to perform the necessary tasks may be stored on a machine-readable medium. A processor may perform the necessary tasks.

The various methods or processes outlined herein may be coded as software executable on one or more processors using any one of a variety of operating systems or platforms. Furthermore, such software may be written using any of a number of suitable programming languages and/or programming or scripting tools, and compiled as executable machine language code or intermediate code that runs on a framework or virtual machine. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

Embodiments of the present disclosure may be embodied as methods, which are provided by way of example. The operations performed as part of the method may be ordered in any suitable manner. Thus, while shown as sequential operations in the exemplary embodiments, embodiments may be constructed in which operations are performed in an order different from that shown, which may include performing some operations simultaneously.

Although the present disclosure has been described with reference to certain preferred embodiments, it should be understood that various other adaptations and modifications may be made within the spirit and scope of the present disclosure. It is therefore the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the present disclosure.

Alternatively, the execution module 206e executes a learning function for state _S1 , once state _S1 is received, with the execution module 206e executing a termination function.

Claims (14)

1. A controller for controlling movement of a robot to perform a task, comprising:
The controller:
an input interface for receiving a current state of the robot for each control step;
[0023] a memory;
the memory is configured to store a library of skills for the robot, each skill being a probability function of a state of the robot that returns a distribution of actions in response to the current state being input from the input interface , the skills being task agnostic;
the memory is configured to store a learned function for the task, the learned function being a probability function of the state of the robot that returns a distribution of skills in response to the state of the robot being input from the input interface ;
The controller further comprises:
a processor configured to execute the learned function for the current state to select, for each control step in a sequence of control steps that reaches an exit condition, a skill having a highest probability on the distribution of skills, and to execute the selected skill for the current state to select an action having a highest probability on the distribution of actions for transitioning the state of the robot from the current state to a next state;
The controller further comprises:
an output interface configured to instruct the robot to perform the selected action to perform the task ;
The learned function is learned based on reinforcement learning having a reward function or learned by expectation maximization;
The controller, wherein the reward function includes a penalty for switching between skills or a penalty based on the distance between the selected action and one or more demonstrated actions, minimizing deviation in the outcome of the execution of the task. 2. The controller of claim 1, wherein parameters of the learned function are selected to maximize, for each possible state, a probability of returning from the library of skills that minimizes a probability of a difference between an action returned by the skill and an action corresponding to multiple executions of the task for the current state , each execution of the task being represented by a sequence of state and action value pairs. The controller of claim 1 , wherein the learned function is a multiple argument function. The controller of claim 1, wherein at least some of the skills are multi-step skills configured to be performed for multiple control steps, and the processor repeats the selection of a skill for each control step even if the currently selected skill is a multi-step skill. The controller of claim 1, wherein the function for selecting skills is learned to penalize switching between skills. The controller of claim 1 , wherein the learned function selects a skill by returning a time to switch a current skill out of a sequence of skills. a learning module configured to learn the function for selecting the skill;
To train the function, the training module comprises:
receiving a set of executions of the task, each execution defining a sequence of state/action pairs;
To train the function, the training module further comprises:
determining, for each state, a distribution of actions from the sequence of state/action pairs that fit the actions received for that state;
and determining, for a state, the skill that has the highest probability of returning an action sampled on the distribution of actions determined for that state. 1. A method for controlling the movement of a robot to perform a task, the method using a processor coupled to a memory, the memory storing (1) a library of skills for the robot, each skill being a probability function of a state of the robot that returns a distribution of actions in response to a current state being input from an input interface , the skills being task agnostic, the memory storing (2) a learned function for the task, the learned function being a probability function of the state of the robot that returns a distribution of skills in response to the state of the robot being input from the input interface , the processor coupled to stored instructions that, when executed by the processor to perform steps of the method,
accepting a current state of the robot for each control step;
executing the learned function on the current state to select, for each control step in a sequence of control steps that reaches a termination condition, the skill that has the highest probability on the distribution of skills;
executing the selected skill for the current state to select an action having the highest probability on the distribution of actions to transition the state of the robot from the current state to a next state;
and outputting a command to the robot to perform the selected action to perform the task;
The learned function is learned based on reinforcement learning having a reward function or learned by expectation maximization;
The method of claim 1, wherein the reward function includes a penalty for switching between skills or a penalty based on the distance between the selected action and one or more demonstrated actions, minimizing deviation in the outcome of the performance of the task. 9. The method of claim 8, wherein parameters of the learned function are selected to maximize the probability of returning, for each possible state, a skill from the library of skills that minimizes the probability of a difference between an action returned by the skill and an action corresponding to multiple executions of the task for the current state , each execution of the task being represented by a sequence of state and action value pairs. The method of claim 8 , wherein the learned function is a multiple argument function. The method of claim 8, wherein at least some of the skills are multi-step skills configured to be performed for multiple control steps, and the processor repeats the selection of a skill for each control step even if the currently selected skill is a multi-step skill. The method of claim 8 , wherein the function is trained to select skills based on a function that penalizes switching between skills. The method of claim 8, wherein the learned function selects the skill by returning the time to switch the current skill out of the sequence of skills. the memory is further configured to store a learning module, which when executed by the processor to perform the steps of the method includes learning the function for selecting the skill;
The learning of the function further comprises:
receiving a set of executions of the task, each execution defining a sequence of state/action pairs;
The learning of the function further comprises:
determining, for each state, a distribution of actions from the sequence of state/action pairs that fit the actions received for that state;
and determining for a state the skill that has the highest probability of returning an action sampled on the distribution of actions determined for that state.

Images