JP7737865B2 - Motion command generating device and motion command generating method - Google Patents

Description

The present invention relates to a motion command generation device and a motion command generation method for generating motion commands for a robot.

In order to improve production efficiency and reduce labor costs, there are increasing efforts to use robots to perform tasks previously performed by humans, such as assembling, welding, and transporting industrial products. However, previous robotic systems required extensive programming and a high level of specialized knowledge, which was an obstacle to the introduction of robots.

To address these situations, an autonomous learning robot control system has been proposed, in which the robot determines its own behavior based on information from various sensors attached to the robot. Compared to previous robot systems, this system does not require extensive programming or advanced specialized knowledge, and is expected to enable robots to be easily introduced. Furthermore, this autonomous learning robot control system is expected to enable the robot to memorize and learn from its own behavioral experiences, enabling it to flexibly generate behavior (generate behavior commands) in response to diverse environmental changes.

A robot's operational experience can be achieved, for example, by the robot's operator/manager directly teaching the robot how to operate and having it memorize it, or by having the operator/manager or another robot observe and imitate the operation. Furthermore, autonomous learning robot control systems are generally equipped with a learning device called a learner, which stores sensor information from operational experience and adjusts parameters to generate operations. These stored operations are called learning data, and parameter adjustment is called learning, and the learner learns using the learning data. The learner defines the input/output relationship in advance and repeatedly learns (adjusts parameters) so that the expected output value is output for the input value to the learner.

An example of learning data is time-series data of a robot's joint angle information and captured images of the task when it experiences a certain action. If the task involves grasping an object, the captured images will show the object being grasped, the robot arm, and the robot hand. Using this learning data, joint angle information and images at time (t) are input into a learning device, which is then trained to predict joint angle information and images at time (t+1). Then, by sequentially inputting robot joint angle information and images into the learning device once learning has completed, the autonomous learning robot control system will be able to automatically generate actions according to the task status.

Such methods generate robot behavior directly from sensor information at a given time, without explicitly going through object recognition, and are therefore known as autonomous learning-based end-to-end behavior generation methods. With these methods, if an object that was the target of work during learning or a similar object is detected in the captured image, the robot will autonomously begin working on that object. However, these methods do not take into account the situation where multiple target objects are present at the same time, which could result in the robot being unable to perform the work or not knowing which object to work on.

On the other hand, there are cases where robots that require autonomy have to work on multiple objects at the same time. Examples of such tasks include removing rubble during decommissioning work, opening and closing valves in a plant, and transporting cardboard boxes. In rubble removal work, rubble of similar shapes and textures is expected to be scattered, and in some cases it is sufficient to remove it in any order, but in other cases rubble overlaps, so it is necessary to remove it in the appropriate order. In addition, in valve opening and closing tasks, the valves must be opened and closed in the appropriate order. For this reason, an autonomous learning-based end-to-end behavior generation method is required that allows a target object to be specified and the robot to perform the task on that object.
A robot device described in Patent Document 1 is a technology for specifying an object to be worked on, in which an operator specifies the position and posture (direction) of an object in an image captured by the robot device.

JP 2013-173209 A

The robot device described in Patent Document 1 plans the activation of an arm to grasp an object whose position and orientation are specified, and controls its movement according to this trajectory. In autonomous learning-based end-to-end movement generation, the robot's movement (movement command) is generated directly without explicit object recognition (recognition of the object's position and orientation). Therefore, it is not possible to specify the position and orientation of an object, as with the robot device described in Patent Document 1.
The present invention has been made in consideration of the above background, and aims to provide an action command generation device and an action command generation method that enable selection of an object in autonomous learning type end-to-end action command generation.

In order to solve the above-mentioned problems, the motion command generation device of the present invention includes an acquisition unit that acquires images and sensor information including candidate objects for a robot to work on, and a command generation unit that uses a machine learning model to input the images and the sensor information and outputs motion commands for the robot, the machine learning model including a position extraction block including parameters that are referenced when calculating the position or area of the candidate object, and a motion command generation block including parameters that are referenced when calculating a motion command for the robot to work on the object, wherein explanatory variables include the images and the sensor information, and a target variable is generated using learning data including the motion commands, and the command generation unit acquires work object designation information that specifies an object to be worked on by the robot from among the candidate objects, and replaces the information on the position or area of the candidate object calculated using the position extraction block with information on the position or area of the object indicated by the work object designation information, and outputs the motion command .

The present invention provides a motion command generation device and a motion command generation method that enable target object selection in autonomous learning-type end-to-end motion command generation. Issues, configurations, and advantages other than those described above will become clear from the description of the following embodiments.

1 is an overall configuration diagram of a robot control system according to a first embodiment. 1 is a functional block diagram of a motion command generating device according to a first embodiment. FIG. 2 is a configuration diagram of a command generation model according to the first embodiment. 10 is a flowchart of an operation command generation process according to the first embodiment. 10 is a photographed image according to a modified example of the first embodiment. 10 is an image for a mask according to a modified example of the first embodiment. A photographed image according to a modification of the first embodiment is a photographed image masked with an image. 10 is a flowchart of an operation command generation process according to the second embodiment. 13 is a flowchart of an operation command generation process according to the third embodiment.

<<Outline of the motion command generation device>>
The following describes a motion command generation device according to an embodiment of the present invention. The motion command generation device receives a captured image including a work object as input and generates motion commands for a robot using a command generation model, which is a trained machine learning model.

The command generation model includes a position extraction block and a motion command generation block. The position extraction block includes parameters used in the process of recognizing one or more work targets (objects) in the captured image. In other words, the position extraction block includes parameters referenced when calculating the position or area of a work target (a candidate work object). The motion command generation block includes parameters used in the process of generating a motion command that instructs the robot to operate to grasp a single work target. In other words, the motion command generation block includes parameters referenced when calculating a motion command for the robot to work on an object.

Note that the command generation model, position extraction block, and action command generation block are parameters used in processes such as recognition and inference using machine learning technology, but below they may be described as if the parameters themselves are the entities that use them for processing. For example, they may be described as "the command generation model calculates the action command" or "the captured image that is input to the position extraction block."

The motion command generation device uses training data to train a machine learning model including a position extraction block and a motion command generation block, generating a command generation model that is a trained machine learning model. The explanatory variables (input) of the training data include captured images, and the objective variables (output, correct answer) include motion commands for a robot to grasp a single object (work target, target object) in the captured images.

The learning data may be instruction-type learning data that includes captured images of a person operating a robot to grasp an object and motion commands that indicate how the robot should be operated when grasping. It may also be learning data in which the optimal grasping motion for an object is calculated without human operation, for example using machine learning technology, and the calculated motion is recorded as a motion command. Note that the number of work target objects shown in the captured images of the instruction-type learning data may be one.

The motion command generation device acquires one or more work objects (position information) from the captured image using the command generation model (position extraction block). Next, the motion command generation device queries the operator to select one work object from the one or more work objects. The motion command generation device then generates a motion command for the robot to grasp the selected work object using the command generation model (motion command generation block). In other words, the motion command generation device replaces the one or more work objects acquired using the command generation model (position extraction block) with a single work object and generates a motion command to grasp the single work object.

This type of motion command generation device uses a command generation model generated using easily obtainable teaching/autonomous learning data to generate a motion command to grasp a work object designated by a worker from among multiple work objects. Previously, when using machine learning models generated using teaching-type learning data, it was not possible to guarantee operation when there were multiple work objects (it was unclear how the object would operate). With this motion command generation device, it is possible to guarantee the order in which the work objects are worked on. For example, it becomes possible to use a robot for work where the work order is fixed.

<Overall configuration of the robot control system>
1 is a diagram showing the overall configuration of a robot control system 10 according to the first embodiment. The robot control system 10 includes a robot 300, a control device 310, cameras 371 and 372, and an operation command generating device 100.
The robot 300 is capable of handling a work target 380, which is an object, and performs predetermined tasks such as assembling and transporting parts. The configuration of the robot 300 is not important, and it may be a robot arm alone, or may be equipped with a moving device such as crawlers or wheels.

The control device 310 is a device that outputs control commands to the robot 300 to control the movement of the robot 300 based on movement commands such as the joint angles of the robot 300 and the posture (position) and force (torque) of the end effector 301 (robot hand) input from the movement command generating device 100. The control commands are signals that indicate, for example, current values and voltage values for actuators (motors, etc.) provided on the joints of the robot arm and the end effector 301 of the robot 300. When the robot 300 receives a control command from the control device 310, its built-in drive circuit supplies a drive signal to the corresponding actuator.

Cameras 371 and 372 are imaging devices for capturing images of the working environment and surrounding environment of robot 300. In FIG. 1, two cameras 371 and 372 are installed, but there may be one or three or more. Camera 371 is attached to robot arm 302, and camera 372 is installed around robot 300 (for example, a workroom or a building wall). Here, the working environment corresponds to the movable range (work area) of robot 300, and the surrounding environment corresponds to the surrounding area outside the movable range of the robot.

The motion command generation device 100 plans the motion of the robot 300 based on images (photographed images) taken by the cameras 371, 372 and information obtained from sensors placed on the robot 300, the work environment, and the surrounding environment (hereinafter referred to as sensor information), and transmits motion commands such as the joint angles and forces of the robot 300 to the control device 310. The type of robot 300 and sensor information is not important here. For example, the sensor information may be the current value of an actuator provided in the joints of the robot 300, or the output signal of a tactile sensor or inertial sensor externally attached to the robot 300. Furthermore, the sensor information may be a temperature sensor measuring the work environment or surrounding environment. In this way, each sensor detects the state of the robot 300 and the state of the environment, and outputs a detection signal according to the detected content.

<Configuration of the motion command generating device>
2 is a functional block diagram of the motion command generating device 100 according to the first embodiment. The motion command generating device 100 is a computer, and includes a control unit 110, a storage unit 120, and an input/output unit 180.
User interface devices such as a display, keyboard, and mouse are connected to the input/output unit 180. For example, a touch panel display is connected to the input/output unit 180, and images captured by cameras 371 and 372 are displayed on the touch panel display. The robot operator designates one of the work objects 380 to be grasped by the robot 300 on the touch panel display. The input/output unit 180 also includes a communication device, and is capable of transmitting and receiving data (signals) to and from devices such as the control device 310 and the cameras 371 and 372.

The memory unit 120 is configured to include storage devices such as ROM (Read Only Memory), RAM (Random Access Memory), and SSD (Solid State Drive). The memory unit 120 stores training data 130, a command generation model 140, and a program 128. The program 128 includes a description of the operation command generation process (see Figure 4, described below).

<Motion command generation device: learning data>
The training data 130 is training data used for training the command generation model 140, which is a machine learning model. The explanatory variables (input) of the training data 130 are photographed images of the work environment and surrounding environment of the robot 300, acquired in time series while the robot 300 is working to grasp the work object 380, and sensor information of the robot 300, the work environment, and the surrounding environment. The objective variables (output) of the training data 130 are operation commands for the robot 300, acquired in time series while the robot 300 is working to grasp the work object 380.

In this task, the operator of the robot 300 connects an operating unit, such as a joystick, to the input/output unit 180 to operate (control) the robot 300. The operating unit is a device that accepts input from the operator and transmits operation commands corresponding to the input to the control device 310. Instead of the operator operating the robot 300, the robot 300 may reproduce pre-planned operations. Note that even if multiple work targets 380 are expected when the robot 300 is in operation, there may only be one work target 380 when the learning data 130 is acquired.

<Motion command generation device: command generation model>
3 is a configuration diagram of the command generation model 140 according to the first embodiment. The command generation model 140 is a machine learning model, and includes a position extraction block 141 and an action command generation block 142.
The position extraction block 141 includes parameters used in the process of recognizing one or more work objects 380 and end effectors 301 in the captured image. The position extraction block 141 includes, for example, parameters related to a convolutional neural network (CNN) that extracts image features and spatial softmax that extracts coordinate information of the position with the strongest intensity from the obtained feature map, and is referenced when performing the process of extracting position information of the work object 380 from the captured image. Spatial softmax is a type of function used in machine learning, such as the softmax function, tanh function, or sigmoid function, and is also called soft argmax. The number of position coordinates extracted by spatial softmax is determined based on the number of channels of the previous CNN.

The motion command generation block 142 contains parameters used in the process of generating motion commands that instruct the robot 300 to grasp the work object. The motion command generation block 142 generates motion commands from the position information and sensor information of the work object obtained from the position extraction block, using, for example, a fully connected layer or a recurrent neural network (RNN).

<<Motion command generating device: control unit>>
2, the description of the control unit 110 will continue. The control unit 110 is configured to include a CPU (Central Processing Unit) and is equipped with an acquisition unit 111, a learning unit 112, and a command generation unit 113. The acquisition unit 111 acquires images captured by cameras 371 and 372, and sensor information from sensors equipped in the robot 300, the work environment, and the surrounding environment.

The learning unit 112 trains the command generation model 140 using the learning data 130 (allows the command generation model 140 to learn the learning data 130). As a result of the training/learning, the parameters included in the command generation model 140 are adjusted, and an action command for grasping the work object 380 that minimizes error based on the captured image and sensor information is output.

Since minimizing the error in the motion commands is equivalent to completing the task indicated in the training data 130, it is expected that important position information for completing the task will be obtained as the output of the position extraction block 141. For example, it is expected that position information of the work object 380 and the end effector 301 will be obtained. Note that the training data 130 includes captured images, sensor information, and motion commands, but does not include position information, and is not explicitly trained/learned.

Furthermore, each channel of the CNN obtained as a result of learning responds to a specific shape. For example, the first channel responds to the work object 380, and the second channel responds to the end effector 301. The motion command generation block 142 predicts motion commands based on the coordinates obtained from each channel, so a change in the coordinates of the first channel is equivalent to a change in the position of the work object. Note that which channel responds to the work object can be determined by inputting the learning data after training/learning and examining the channel that is outputting the position coordinates of the work object. In this way, the position information output by the position extraction block 141 includes the position coordinates of each work object 380 and end effector 301 included in the captured image.

The command generation unit 113 uses the command generation model 140 to make inferences (predictions) based on the captured image, sensor information, and position information of the work object specified by the operator, and outputs operation commands for the robot 300. These operation commands are sent to the control device 310, causing the robot 300 to operate.

The position information of the work object specified by the operator is not an input (explanatory variable) to the command generation model 140, which is a machine learning model. The command generation unit 113 replaces the position information of the work object calculated using the position extraction block 141 with the position information of the work object specified by the operator, and calculates an action command using the action command generation block 142.

<<Motion command generation process>>
4 is a flowchart of the motion command generation process according to the first embodiment. In the following description of the motion command generation process, the main task is the task of the robot 300 grasping one or more work objects 380 (e.g., rubble) one by one with the end effector 301 and transferring (removing) them to a predetermined container (not shown). Grasping one work object 380 and transferring it to the container is one sub-task. In the first embodiment, the operator of the robot 300 indicates the work object 380 to be removed, and the robot 300 grasps the instructed work object 380 and transfers it to the container in accordance with the generated motion command.

In step S11, the command generation unit 113 begins the process of repeating steps S12 to S17 for each sub-task. In more detail, the command generation unit 113 uses the position extraction block 141 to calculate position information from the captured image, and if the position information does not include position information for the work object 380, the command generation process ends; if the work object 380 is present, the command generation unit 113 proceeds to step S12.

In step S12, the command generation unit 113 displays images captured by cameras 371 and 372 showing the work objects 380 on the touch panel display and asks the operator which work object 380 to grasp. When the operator touches and indicates one of the work objects 380 to be grasped, the command generation unit 113 acquires its position. In other words, the command generation unit 113 acquires work object designation information that specifies the object to be worked on by the robot 300 from among the candidate objects to be worked on. Hereinafter, the one designated work object 380 will be referred to as the work object.

In step S13, the command generation unit 113 repeats the processes of steps S14 to S17 until the sub-task is completed (the robot grasps the work object instructed in step S12 and transfers it to a container).
In step S14, the command generation unit 113 acquires position information by inputting the captured image to the position extraction block 141. More specifically, the command generation unit 113 calculates position information from the captured image using the position extraction block 141. The position information includes the position coordinates of each work object 380 and the end effector 301.

In step S15, the command generation unit 113 replaces the position coordinates of the work target object included in the position information with the position information acquired in step S12. Note that the position of the work target object changes as the command generation unit 113 repeats steps S14 to S17 and the robot 300 grasps and moves the work target object. The command generation unit 113 tracks the position of the work target object instructed in step S12 and replaces the position with the position of the work target object that is the tracking result. The position of the work target object after grasping can be obtained from the position of the end effector 301 calculated from the operation command acquired in step S16, which will be described later.

In step S16, the command generation unit 113 inputs the replaced position information and sensor information to the action command generation block 142 to obtain an action command. Specifically, the command generation unit 113 calculates an action command from the replaced position information and sensor information using the action command generation block 142.
In step S17, the command generator 113 transmits the operation command calculated in step S16 to the control device 310.

<Features of the motion command generation device>
The motion command generation device 100 generates motion commands for the robot 300 from the captured image and sensor information using the command generation model 140. The command generation model 140 is a machine learning model, and its learning data is data that can be acquired from the work performed by the operator operating the robot 300, and can be created at low cost.

The command generation model 140 includes a position extraction block 141 and a motion command generation block 142, and is configured to calculate the position of the work target when generating a motion command. When generating a motion command, the motion command generation device 100 replaces the position of the work target with the position of the work target specified by the operator. In this way, even if there are multiple objects to be worked on, the robot 300 will perform the work on the object specified by the operator.
The number of work objects to be processed when the learning data is generated (when the operator operates the robot 300) may be one. Compared to preparing learning data for multiple work objects and then training/learning, the learning data can be generated at low cost and in a short time.

<<Variation: Location Information>>
In the first embodiment described above, position information (position coordinates) of the work object is calculated (see FIG. 3 ) when calculating an action command using the command generation model 140. By obtaining a heat map using a sigmoid function instead of the Spatial Softmax function in the position extraction block 141, area information of the work object and the end effector 301 may be calculated instead of position information.

<<Variation: Replacing location information>>
In the first embodiment described above, the command generator 113 replaces the position information of the work object output from the position extraction block 141 with the position information of the work object specified by the operator (see step S15 in FIG. 4). The captured image to be input may include only the work object.

Figure 5 shows a captured image 510 according to a modified example of the first embodiment. Suppose the operator specifies the right work object 511 to be grasped among the three work objects at the bottom right of the captured image 510. The command generation unit 113 then generates a masking image 520 (see Figure 6 described below), which is an image that masks the area including work object 511 and the rest, and which masks all work objects other than work object 511 that the operator did not specify.

Figure 6 shows a mask image 520 according to a modified example of the first embodiment. Area 521 is an area that includes work object 511 and does not include work objects other than work object 511 that the operator has not specified. In image 520, the area other than area 521 is masked.

Figure 7 shows a captured image 530 in which captured image 510 according to a modified example of the first embodiment has been masked with image 520. Rather than replacing the position information, the command generation unit 113 inputs the captured image 530, in which the specified work object 511 remains and the rest is masked, into the command generation model 140 (position extraction block 141) to calculate an action command. By inputting an image in which only the work object 511 is captured, the position information output by the position extraction block 141 includes only the position information of the work object 511, and an action command to grasp the work object 511 is generated.

<<Variation: Selection of work type>>
In the first embodiment described above, the operator selects the work target (see step S12 in FIG. 4 ), but the operator may also select one of a plurality of work types. The work type in the rubble removal work refers to, for example, the container to which the rubble of the work target that has been selected and grasped by the robot 300 is to be transferred. Similarly, the work type in the valve opening/closing work refers to, for example, whether the selected valve is to be opened or closed.

There are multiple command generation models 140 corresponding to different task types, and each is trained/learned to generate operational commands corresponding to the task type. In step S12, the command generation unit 113 acquires the task type along with the position information of the task object. In steps S14 to S17, the command generation unit 113 calculates operational commands using the command generation model 140 corresponding to the task type.
In this way, the robot 300 can perform the work type instructed by the operator on the work object.

Second Embodiment
In the first embodiment described above, the operator is queried about the work object to be grasped (see step S12 in FIG. 4). If it is possible to identify a single work object to be grasped, the work may be performed without querying the operator. For example, if the work object can be considered to be in a single location in the captured image, the work may be performed without querying the operator. This is the case when there is more than one piece of position information (position coordinates) calculated by the position extraction block 141, but the dispersion (variance) of the positions is small and the positions can be considered to be in a single location.

The functional configuration of the motion command generation device 100 according to the second embodiment is the same as that of the first embodiment, except for the command generation unit 113 (motion command generation processing). The command generation unit according to the second embodiment is referred to as a command generation unit 113A. Fig. 8 is a flowchart of the motion command generation processing according to the second embodiment.
In step S31, the command generator 113A starts the process of repeating steps S32 to S40 for each sub-task.
In step S32, the command generator 113A inputs the captured image to the position extraction block 141 to acquire position information and calculates the variance of the position information. The variance is, for example, the sum of the variances of the X and Y coordinates that indicate the position of the work object.

In step S33, if the variance calculated in step S32 is greater than a predetermined value (step S33→YES), the command generator 113A proceeds to step S34, and if it is equal to or less than the predetermined value (step S33→NO), the command generator 113A proceeds to step S35.
Step S34 is the same process as step S12 in FIG.
In step S35, the command generation unit 113A repeats the processes of steps S36 to S40 until the robot grasps the work target object instructed in step S34 and transfers it to the container. If step S34 is skipped (step S33→NO), the work target object is an object that is located at a position with position information that has small variance and can be considered as a single location.

Step S36 is the same process as step S14.
In step S37, if the command generation unit 113A has acquired the position of the object to be worked on, which is the operator's instruction, in step S34 (see step S12) (step S37 → YES), it proceeds to step S38, and if it has not acquired the position (step S37 → NO), it proceeds to step S39.
Steps S38 to S40 are the same processes as steps S15 to S17, respectively.

Features of the Second Embodiment
If the work object can be regarded as being in one place in the captured image, the robot 300 performs the work without asking the operator, thereby improving work efficiency.
Alternatively, step S38 may be executed immediately after step S36 without skipping step S38 by branching to NO in step S37. In this case, in step S38, the command generator 113A may replace the position coordinates of the work target object with position information corresponding to the movement of the gripped work target object (see step S15).

Third Embodiment
In the first embodiment, the order of work objects is fixed. If the order is not fixed, the work may be performed one after another without asking the operator. For example, in a rubble removal work, if rubble of similar shape and appearance is scattered and can be removed in any order, the rubble may be picked up one after another and placed in a container.

The functional configuration of the motion command generation device 100 according to the third embodiment is the same as that of the first embodiment, except for the command generation unit (motion command generation process). The command generation unit according to the third embodiment is referred to as a command generation unit 113B. Fig. 9 is a flowchart of the motion command generation process according to the third embodiment.
In step S51, the command generator 113B starts the process of repeating steps S52 to S58 for each sub-task.
In step S52, the command generation unit 113B inputs the captured image to the position extraction block 141 to obtain position information of the work object.

In step S53, the command generation unit 113B randomly selects one of the work targets in the position information acquired in step S52. Hereinafter, this selected work target will be referred to as a work target object.
Steps S54 to S58 are similar to steps S13 to S17, respectively, except that in step S56, the position coordinates of the work target object included in the position information are replaced with the position information of the work target object selected in step S53.

Features of the Third Embodiment
The robot 300 performs the work without asking the operator, thereby improving work efficiency.

<<Variations>>
Although several embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. For example, while the removal of rubble (grasping the rubble and moving it to a container) has been described as an example of the task of the robot 300, other tasks may be performed. For example, tasks may include opening and closing multiple valves in a plant in a predetermined sequence, or transporting objects (boxes). In the case of the task of opening and closing valves, a sub-task is the opening and closing of one valve.

In the first embodiment described above, the operator of the robot 300 selects the object to be worked on, but this is not limiting. There is a system that determines the order of the objects to be worked on, and this system may select the objects to be worked on.
In the first embodiment described above, the command generator 113 determines whether or not the work object 380 (position information) is present based on the position information (see step S11 in FIG. 4), but the operator may also determine this based on the captured image. Also, the operator may also determine the end of the sub-task (see step S13).

The motion command generation device 100 in the above embodiment includes a learning unit 112 and command generation units 113, 113A, and 113B, generates a command generation model 140, and outputs motion commands using the command generation model 140. The device that generates the command generation model 140 and the device that outputs the motion commands may be separate. For example, a model generation device including a learning unit 112 may generate the command generation model 140 and transmit it to multiple motion command devices, and each motion command device may output a motion command to the control device of each robot. Alternatively, the motion command generation device 100 and the control device 310 may be integrated.

The present invention may take on various other embodiments, and various modifications, such as omissions and substitutions, may be made without departing from the spirit of the present invention. These embodiments and their variations are included within the scope and spirit of the invention described in this specification, etc., and are also included in the scope of the invention described in the claims and their equivalents.

100 Operation command generating device 111 Acquisition unit 112 Learning unit 113, 113A, 113B Command generating unit 128 Program 130 Learning data 140 Command generation model (machine learning model)
141 Position extraction block 142 Operation command generation block 300 Robot 380 Work object (object)

Claims (7)

an acquisition unit that acquires images and sensor information including candidates for objects on which the robot will work;
a command generation unit that uses a machine learning model to input the image and the sensor information and outputs an operation command for the robot;
The machine learning model is
a position extraction block including parameters to be referenced when calculating a position or area of the candidate object; and an operation command generation block including parameters to be referenced when calculating an operation command for the robot to perform a task on the object,
an explanatory variable including the image and the sensor information, and a target variable including the operation command , the explanatory variable being generated using learning data;
The command generation unit
acquiring work object designation information that designates an object to be worked on by the robot from among the candidate objects;
The information on the position or area of the candidate object calculated using the position extraction block is replaced with the information on the position or area of the object indicated by the work object designation information, and the operation command is output.
1. A motion command generating device comprising: The machine learning model is
Multiple machine learning models for each type of work,
The command generation unit
acquiring the type of work to be performed by the robot on the object corresponding to the work object designation information;
The operation command generating device according to claim 1, characterized in that a machine learning model corresponding to the task type among the plurality of machine learning models is used to input the image and the sensor information and output an operation command for the robot. The command generation unit
The operation command generation device according to claim 1, wherein the machine learning model is used to input an image in which an area different from the area of the object indicated by the work object designation information is masked, and the operation command is output. The command generation unit
When the variance calculated from the information on the position or area of the candidate object calculated using the position extraction block is greater than a predetermined value,
2. The action command generating device according to claim 1 , wherein information on the position or area of the candidate object calculated using the position extraction block is replaced with information on the position or area of the object indicated by the work target designation information, and the action command is output. The command generation unit
The action command generating device according to claim 1, characterized in that the image is displayed on a display device, and information on a position or area in the image of a candidate object designated from among the displayed candidate objects is used as the work object designation information. The command generation unit
selecting one of the position or area information of the candidate object calculated using the position extraction block as the work object designation information;
2. The action command generating device according to claim 1, wherein information on the position or area of the candidate object calculated using the position extraction block is replaced with information on the position or area of the object indicated by the work target designation information, and the action command is output. The motion command generating device
acquiring image and sensor information including candidate objects for the robot to work on;
and using a machine learning model, outputting an operation command for the robot using the image and the sensor information as input.
The machine learning model is
a position extraction block including parameters to be referenced when calculating a position or area of the candidate object; and an operation command generation block including parameters to be referenced when calculating an operation command for the robot to perform a task on the object,
an explanatory variable including the image and the sensor information, and a target variable including the operation command , the explanatory variable being generated using learning data;
In the step of outputting the operation command,
acquiring work object designation information that designates an object to be worked on by the robot from among the candidate objects;
The information on the position or area of the candidate object calculated using the position extraction block is replaced with the information on the position or area of the object indicated by the work object designation information, and the operation command is output.
A motion command generating method comprising: