JP7811524B2 - A system that outputs control information to control a robot hand that can change the insertion depth into a collection area for small, irregularly shaped ingredients. - Google Patents

Description

The present invention relates to a system that outputs control information for controlling a robotic hand that can change the insertion depth into a collection area for small, irregularly shaped ingredients.

Conventionally, the task of placing small, irregularly shaped ingredients, such as green onions, into lunch boxes and other such items has been carried out by a person manually removing the small, irregularly shaped ingredients from a storage area for small ingredients, such as a food tray, by grasping them with their fingers and placing them in the appropriate position in the lunch box or other such item.

However, labor shortages are expected to become more serious in the near future. For this reason, the introduction of robots to automate each process is being considered. Automation of food plating is also being considered as part of this effort.

For example, as shown in Patent Document 1, a robotic hand (soft gripper) that grasps ingredients has been developed to automate the process of plating ingredients.

Japanese Patent Application Laid-Open No. 2020-168691

Girshick, R. B.: Fast R-CNN, CoRR, Vol. abs/1504.08083(2015). Ren, S., He, K., Girshick, R. and Sun, J.: Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, No. 6, pp. 1137-1149 (2017).

However, when attempting to grasp and remove small, irregularly shaped ingredients using a robotic hand (soft gripper) such as that shown in Patent Document 1, the stacking condition of the small, irregularly shaped ingredients piled up in the collection area is not constant (for example, the surface irregularities vary), and the stacking condition changes each time ingredients are removed, making it difficult to consistently remove the desired amount of small ingredients with high precision.

The inventors of this invention have been working diligently to determine whether it is possible to consistently dispense the desired amount of small ingredients with high accuracy by analyzing images of the accumulation area of small, irregularly shaped ingredients and estimating the loading status of the small ingredients based on those images.

In the course of this research, we discovered that in order to realize a highly flexible robotic hand system, it would be effective to (1) use a robotic hand that can change its "insertion depth" into the collection area for small, irregularly shaped ingredients, and (2) have a system that outputs control information for controlling such a robotic hand determine the slope and intercept of a linear function that shows the relationship between the insertion depth of the robotic hand into small ingredients and the weight of the small ingredients that the robotic hand can grasp and remove.

This means that even when the weight of the small ingredients to be removed changes relatively frequently, such as when producing a variety of bento boxes in small quantities, the variable "insertion depth" of the robot hand can be used to achieve more flexible ingredient removal control.

The present invention was developed based on the above findings. The object of the present invention is to provide a system that outputs control information for controlling a robotic hand that can change the insertion depth into a collection location for small, irregularly shaped ingredients, thereby enabling the removal of a desired amount of small ingredients with greater precision.

The present invention is a system that outputs control information for controlling a robotic hand that can change its insertion depth into a collection area for small, irregularly shaped ingredients. The system acquires an image of the collection area for small ingredients, evaluates the similarity with a sample local area, and determines at least one candidate local area from the image. Based on information associated with the sample local area regarding the insertion depth of the robotic hand into the small ingredient and the weight of the small ingredient that can be grasped and removed by the robotic hand, the system determines the slope and intercept of a linear function associated with each candidate local area that indicates the relationship between the insertion depth of the robotic hand into the small ingredient and the weight of the small ingredient that can be grasped and removed by the robotic hand. The system then links the slope and intercept of the linear function associated with each candidate local area to the coordinate data of each candidate local area and outputs them as control information.

The system of the present invention can determine a linear function between the depth of insertion of the robot hand into the small food item and the weight of the small food item that can be removed from an image of the collection location of the small food items. This makes it easier to respond to changes in the weight of the small food item to be removed.

In the present invention, the number of "sample local regions" may be one or more. "Evaluating the similarity with the sample local region" may mean evaluating the similarity by extracting certain parameters using an image processing program or the like and comparing those parameters, or may directly evaluate the similarity (correlation) between the two images using an AI-based learning program or the like.

In addition, in the present invention, "information associated with the sample local area relating to the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand" may refer to the "slope and intercept of a linear function that shows the relationship between the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand" if such information is provided in advance for the sample local area. For example, if there are multiple "sample local areas" that can be determined to have some common characteristics, it is possible to obtain the "slope and intercept of the linear function" using the least squares method based on the "insertion depth" and "grasping weight" of the robot hand that are actually measured for each of them.

In addition, in the present invention, "based on information on the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand, which is linked to the sample local area, determine the slope and intercept of a linear function that indicates the relationship between the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand, which is linked to each candidate local area" means "based on information on the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand, which is linked to the sample local area." If the "information on the weight of the small food ingredient that can be grasped and removed by the robot hand" is the "slope and intercept of a linear function," this includes determining the "slope and intercept of the linear function" as the "slope and intercept of the linear function" that corresponds to (is linked to) each "candidate local region," or determining the "slope and intercept of the linear function" after performing a correction process based on the similarity (degree of correlation) between the "sample local region" and each "candidate local region" as the "slope and intercept of the linear function" that corresponds to (is linked to) each "candidate local region."

Alternatively, in the present invention, "determining the slope and intercept of a linear function associated with each candidate local region that indicates the relationship between the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand, based on information associated with the sample local region regarding the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand" includes, in cases where "information associated with the sample local region regarding the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand" is not provided as "the slope and intercept of a linear function," but is provided, for example, as pairs (sets) of the "insertion depth" and "grasping weight" of the robot hand that are actually measured for each of multiple sample local regions, directly calculating the "slope and intercept of a linear function" that corresponds to (is associated with) "each candidate local region" using, for example, machine learning model data (and an AI program) constructed using these multiple pairs (sets) as training data.

In addition, in this invention, an image may be a D image (distance image, depth image), an RGB image, or an RGBD image that combines the characteristics of both.

Furthermore, in the present invention, when determining or calculating the "slope and intercept of a linear function" corresponding to (associated with) each "candidate local region," it is preferable to also evaluate the "likelihood" of such determination or calculation.

This "likelihood" may be calculated or evaluated based on the similarity (degree of correlation) between the "sample local region" and each "candidate local region," or may be evaluated directly by the AI learning program.

In such cases, the present invention can be defined as follows. That is, one aspect of the present invention is a system that outputs control information for controlling a robotic hand that can change its insertion depth into a collection area for small, irregularly shaped ingredients. The system acquires an image of the collection area for small ingredients, evaluates similarity with a sample local area to determine at least one candidate local area from the image, and determines the slope, intercept, and likelihood of a linear function associated with each candidate local area that indicates the relationship between the insertion depth of the robotic hand into the small ingredient and the weight of the small ingredient that can be grasped and removed by the robotic hand based on information associated with the sample local area regarding the insertion depth of the robotic hand into the small ingredient and the weight of the small ingredient that can be grasped and removed by the robotic hand. The system outputs the slope, intercept, and likelihood of the linear function associated with each candidate local area as control information, linking them to the coordinate data of each candidate local area.

In this aspect, if there are multiple candidate local regions, it is more preferable to determine the candidate local region with the highest likelihood as the final candidate local region, and to link the slope and intercept of the linear function associated with the final candidate local region to the coordinate data of the final candidate local region and output them as control information.

The present invention can also be defined as a method. Specifically, this method invention is a method for outputting control information for controlling a robotic hand capable of changing the insertion depth into a collection location for small, irregularly shaped ingredients, comprising the steps of: acquiring an image of the collection location for small ingredients; determining at least one candidate local region from the image by evaluating the similarity with a sample local region; and determining the slope and intercept of a linear function associated with each candidate local region that indicates the relationship between the insertion depth of the robotic hand into the small ingredient and the weight of the small ingredient that can be grasped and removed by the robotic hand, based on information associated with the sample local region regarding the insertion depth of the robotic hand into the small ingredient and the weight of the small ingredient that can be grasped and removed by the robotic hand; and linking the slope and intercept of the linear function associated with each candidate local region to the coordinate data of each candidate local region and outputting them as control information.

When "likelihood" is also evaluated, the method invention is a method for outputting control information for controlling a robotic hand that can change its insertion depth into a collection location for small, irregularly shaped ingredients, comprising the steps of: acquiring an image of the collection location for small ingredients; determining at least one candidate local area from the image by evaluating the similarity with a sample local area; and determining the slope, intercept, and likelihood of a linear function associated with each candidate local area that indicates the relationship between the insertion depth of the robotic hand into the small ingredient and the weight of the small ingredient that can be grasped and removed by the robotic hand, based on information associated with the sample local area regarding the insertion depth of the robotic hand into the small ingredient and the weight of the small ingredient that can be grasped and removed by the robotic hand; and linking the slope, intercept, and likelihood of the linear function associated with each candidate local area to the coordinate data of each candidate local area and outputting them as control information.

The present invention can also be defined as a program for outputting control information for controlling a robotic hand that can change the depth of insertion into a collection location for small, irregularly shaped ingredients, characterized in that the program, when executed by a computer, can carry out the aforementioned method (such programs are also protected by this application).

According to the present invention, a linear function can be determined between the depth of insertion of the robot hand into the small food ingredient and the weight of the small food ingredient that can be removed from an image of the accumulation location of the small food ingredients. This makes it easier to respond to changes in the weight of the small food ingredient to be removed.

1 is a schematic perspective view of a robot hand system including a system according to a first embodiment of the present invention. FIG. FIG. 2 is a schematic perspective view showing a state in which the robot hand of the robot hand system of FIG. 1 is positioned above a collection location. FIG. 2 is a schematic perspective view showing a state in which a robot hand of the robot hand system of FIG. 1 is inserted into a small food ingredient in a collection location. FIG. 2 is a perspective view of a soft gripper of the robot hand of FIG. 1. 2 is a side view of the outer part of the scoop element on which the protruding element is erected in the soft gripper of FIG. 1. FIG. 6 is a perspective view of an outer part of the scoop element on which the protruding element of FIG. 5 is erected. FIG. FIG. 2 is a conceptual diagram illustrating the module configuration of the robot hand system of FIG. 1. FIG. 2 is a conceptual diagram illustrating the general structure of a neural network used in the system in FIG. 1. 2 is a diagram illustrating how pseudo teacher data is assigned to an area where teacher data is unknown in the system in FIG. 1 . FIG. FIG. 2 is a diagram showing the distribution of insertion depth and grip weight in training data for the system in FIG. 1 . 1. FIG. 4 is a diagram showing an example of an RGB image obtained by photographing a small food ingredient (green onion) at a collection location using a camera in the system shown in FIG. 1. FIG. 2 is a diagram showing an example of a D image (depth image) obtained by photographing a small food ingredient (green onion) at a collection location using an RGBD camera in the system shown in FIG. FIG. 2 is a diagram showing an example in which multiple regions are proposed in the system in FIG. 1; FIG. 2 is a diagram showing an evaluation of estimated weight relative to actual weight in an area where training data is known in the system of FIG. 1 . FIG. 10 is a conceptual diagram illustrating the module configuration of a robot hand system including a system according to a second embodiment of the present invention.

The following describes an embodiment of the present invention with reference to the drawings.

(Overall structure)
FIG. 1 is a schematic perspective view of a robot hand system 2 including a system 1 according to a first embodiment of the present invention.

System 1 is a system that outputs control information for controlling the robot hand 8. The robot hand 8 is capable of changing the depth to which irregularly shaped small ingredients 4 are inserted into the accumulation area 6 (see Figure 10). Details of System 1 will be described later.

The robotic hand system 2 is a robotic hand system that grasps and removes small, irregularly shaped ingredients 4 from a collection area 6 for the small ingredients 4. For example, it may also be called a topping device. The robotic hand system 2 includes a camera 10 that captures an image of the collection area 6 for the small ingredients 4 and transmits it to the system 1, and a robotic hand 8 that is controlled based on control information output from the system 1.

The irregularly shaped small food ingredients 4 are, for example, finely chopped green onions or corn, and the collection location 6 is, for example, a food container.

The robot hand 8 is attached to a robot arm 12 (e.g., a duAro manufactured by Kawasaki Heavy Industries, Ltd.). The robot hand 8 is equipped with a soft gripper 14 (see Figure 4) that can gently grasp small ingredients 4. The soft gripper 14 is designed to gently (without damaging) finely chopped ingredients such as green onions or corn, and is equipped with a base member 16 that is positioned above the object to be grasped (see Patent Document 1).

The base member 16 has a generally cross-shaped member 16a and four quadrant arc plates 16b fixed to the underside of the generally cross-shaped member 16a. A circular robot arm mounting portion 13 is attached to the upper surface of the generally cross-shaped member 16a via four support posts 17.

A flexible silicone scoop element 20 hangs vertically downward from the underside of each of the quadrant arc plates 16b of the base member 16. The four scoop elements 20 are arranged adjacent to each other in the circumferential direction in a plan view. Each scoop element 20 has a two-stage tapered shape.

As shown in Figures 4 to 6, nine protruding elements 31 to 39, each with an internal air chamber 41 to 49, are symmetrically erected on the outer surface of each scoop element 20. More specifically, the scoop element 20 is constructed by adhering the scoop element outer portion 20a, on which the nine protruding elements 31 to 39 are erected, to the scoop element inner portion 20b.

The air chambers 41-49 all extend through the scoop element outer portion 20a and into the interior of each protrusion element 31-39. The upper surface of the scoop element outer portion 20b, to which the scoop element outer portion 20a is adhered, is flat.

Air chambers 41-44 formed by protruding elements 31-34 (and scoop element outer portion 20a) are horizontal air chambers that extend parallel to one another in the circumferential direction (horizontal direction), while air chambers 45-49 formed by protruding elements 35-39 (and scoop element outer portion 20a) are vertical air chambers that extend parallel to one another in the vertical direction in the region above the horizontal air chambers 41-44. All air chambers 41-49 are commonly connected by vertical compressed air supply passage 20c and horizontal compressed air supply passage 20d formed in the scoop element outer portion 20a.

The camera 10 is attached to the robot arm 12 so that it can take pictures downward. Therefore, for example, by controlling the robot arm 12, the camera 10 is positioned directly above the center of the collection area 6 when taking pictures.

The camera 10 photographs the stacked state (stacked state) of the small ingredients 4 in the collection area 6 from above to below. In this embodiment, the camera 10 photographs the stacked state (stacked state) of the small ingredients 4 in the collection area 6 from above to below. In this embodiment, the camera 10 photographs the stacked state of the small ingredients 4 from above to diagonally below the collection area 6. The camera 10 in this embodiment is an RGBD camera, and acquires an RGB image corresponding to each pixel in the image, and a distance image (depth image) to the small ingredients corresponding to each pixel in the image.

(Configuration of System 1 of First Embodiment)
The system 1 will now be described in more detail with reference to FIG.

System 1 is configured to realize so-called AI-based machine learning, and as shown in FIG. 7, has a robot control unit 54, a recognition unit 56, and a learning unit 58. The robot control unit 54, recognition unit 56, and learning unit 58 are connected to each other via a wireless communication network such as the Internet (although they may also be connected via a wired connection or built on a single shared server).

In Figure 7, the robot control unit 54, recognition unit 56, and learning unit 58 are shown separately, but some or all of these may form a common functional unit (control unit).

The robot control unit 54 includes, for example, an overall control node 60 that controls the entire system, an information recording node 62 that receives instructions to record data from the overall control node 60, a robot hand control node 64 that receives control instructions for the robot hand 8 from the overall control node 60, a robot arm control node 66 that receives control instructions for the robot arm 12 from the overall control node 60, an image recognition unit management node 68 that receives instructions for the target weight of small ingredients from the overall control node 60, and an RGBD camera control node 70 that controls the camera 10.

The overall control node 60 is connected to an input unit (not shown) and receives commands such as target weight from the input unit. The overall control node 60 is also connected to an output unit (not shown) and is capable of outputting (e.g., displaying) the control information of the overall control node 60 to the outside.

The robot hand control node 64 controls the robot hand 8 and soft gripper 14, and also acquires information on their control status, control position, etc.

The robot arm control node 66 controls the robot arm 12 and acquires information such as its control status and control position.

The RGBD camera control node 70 receives control commands for the camera 10 (RGBD camera) from the overall control node 60. The RGBD camera control node 70 also controls the camera 10 and acquires information such as images captured by the camera 10.

The image recognition unit management node 68 is configured to receive RGBD images (e.g., 100 cm x 55 cm, 1280 pixels x 720 pixels) from the RGBD camera control node 70. The image recognition unit management node 68 also has functions such as preparing image data for image processing in the image recognition node 72 (described below) and converting image coordinates. The image recognition unit management node 68 is configured to synchronize data with the overall control node 60.

The recognition unit 56 has, for example, an image recognition node 72 that mainly performs image processing.

The image recognition node 72 is configured to receive, for example, an RGBD image and target weight from the image recognition unit management node 68.

The image recognition node 72 then evaluates the similarity of the image received from the image recognition unit management node 68 with a pre-selected or set sample local region, and determines at least one candidate local region from the image (for example, 5 cm x 5 cm, or 90 pixels x 90 pixels, regions that are offset from each other in pixel units, i.e., may partially overlap each other). The image recognition node 72 also determines the slope and intercept of a linear function associated with the at least one candidate local region based on information associated with the sample local region, namely, information relating to the insertion depth of the robot hand 8 into a small food ingredient and the weight of a small food ingredient that can be grasped and removed by the robot hand 8. The slope and intercept of the linear function represent the relationship between the insertion depth of the robot hand 8 into a small food ingredient and the weight of a small food ingredient that can be grasped and removed by the robot hand 8.

The image recognition node 72 is configured to synchronize data with the image recognition unit management node 68.

In this embodiment, the image recognition node 72 evaluates the likelihood of the slope and intercept of the linear function corresponding to each candidate local region when determining them (in this embodiment, this is evaluated directly by the AI learning program).

In addition, in this embodiment, the image recognition node 72 transmits to the image recognition unit management node 68, for each of the multiple candidate local regions, its coordinates, the insertion depth relative to the target weight calculated from the slope and intercept of the determined linear function, and its likelihood.

The image recognition unit management node 68 then selects the most likely candidate local region from the received candidate local regions and transmits its coordinates and the corresponding insertion depth to the overall control node 60.

The learning unit 58 has, for example, a machine learning node 74 that performs machine learning.

The machine learning node 74 is capable of acquiring all or part of the information held by the image recognition unit management node 68, such as image data, image coordinates, and insertion depth, as learning data.

In addition, the machine learning node 74 is capable of acquiring all or part of the information held by the overall control node 60, such as the coordinates, target weight, and insertion depth of the robot hand 8, as learning data (training data) for updating.

In addition, the machine learning node 74 can send the machine learning model data (or the updated machine learning model data if there is an update) to the image recognition node 72, allowing it to be used for image processing in the image recognition node 72.

The machine learning node 74 is configured to send and receive data asynchronously with each of the image recognition unit management node 68, image recognition node 72, and information recording node 62.

(Control operation of system 1)
Next, the control operation of the system 1 in this embodiment will be described.

The robot hand system 2 uses the camera 10 to acquire an RGBD image of the location (box) where the small ingredients (green onions) are collected.

The image recognition node 72 acquires the RGBD image from the RGBD camera control node 70 via the image recognition unit management node 68.

Next, the image recognition node 72 of this embodiment receives machine learning model data from the machine learning node 74, evaluates the similarity (correlation) with images of multiple sample local areas (sample local areas (teaching data)) that have been set (stored) in advance, and determines at least one candidate local area from the image. It also calculates the "slope and intercept of a linear function" that directly corresponds to (is linked to) each candidate local area using the machine learning model data (and AI program) constructed by using pairs of the "insertion depth" and "grasping weight" of the robot hand that are actually measured for each of the multiple sample local areas as teaching data, and also calculates the likelihood of the "slope and intercept of a linear function" that corresponds to (is linked to) each candidate local area. (The "multiple sample local regions" themselves can be set based on a neural network (NN) model constructed through machine learning (deep learning) using training data. The machine learning node 74 may store raw data (trainer data) of pairs of the robot hand's "insertion depth" and "grasp weight" measured for each of the multiple sample local regions, but it is sufficient to store only the machine learning model data constructed from the trainer data.)

Next, for each of the multiple candidate local regions, the image recognition node 72 transmits to the image recognition unit management node 68 its coordinates, the insertion depth relative to the target weight calculated from the slope and intercept of the determined linear function, and its likelihood.

The image recognition unit management node 68 then selects the most likely candidate local region from the received candidate local regions and transmits its coordinates and the corresponding insertion depth to the overall control node 60.

The control information output from the image recognition node 72 (e.g., coordinates of determined candidate local regions, information on corresponding linear functions, etc.) may be output to an output unit such as a monitor, either via or without the overall control node 60.

Next, the overall control node 60 lowers the soft gripper 14 of the robot hand 8 and causes it to grasp a green onion (a small, irregularly shaped food ingredient).

When gripping the green onion, the tip 20t of each scoop element 20 of the soft gripper 14 of the robot hand 8 is inserted to a predetermined insertion depth D (see Figure 10). Insertion depth D is, for example, the insertion depth from the surface (average height surface) of the green onion in the candidate local region.

After the tip 20t of the soft gripper 14 is inserted into the green onion, a compressed air supply device (not shown) supplies compressed air to the air supply path 52 of the compressed air inlet 51 of each scoop element 20. This causes each air chamber 41-49 to expand and deform with the compressed air, and in particular the expansion of the lateral air chambers 41-44 causes each scoop element 20 to bend and deform, bringing the tip ends 20t of each scoop element 20 closer to each other. This allows even small, irregularly shaped ingredients such as green onions to be gripped (scooped up) without spilling and transported to the desired container (or other desired destination).

(Machine Learning)
Next, a method for determining (proposing) candidate local regions using AI machine learning in the image recognition node 72 of the system 1 will be described in more detail.

(Machine learning: Relational equation between grasping motion and grasping weight)
First, the relational expression between the gripping action and the gripped weight will be explained.

Let a be the parameter representing the movement of the robot hand when grasping a small food ingredient, and let w be the weight grasped as a result of that movement.

Furthermore, if the local conditions of the small food ingredients to be grasped are the same or similar, it is assumed that the grasping actions also have the same or similar relationship (at least some correlation), and therefore the relationship between a and w can be expressed as follows:
Formula (1)

Here, I _local is a variable representing the local condition of a small food ingredient obtained from an RGBD image, θ(I _local ) is a function value (parameter) determined based on the local condition, and f is a function that takes the function value as one of its inputs.

In an actual grasping operation, factors that are not expressed as I _local or a, such as subtle deformations of the robot hand, errors contained in the robot hand's operation itself, and the internal conditions of the food ingredient, are also thought to affect the grasping weight w, so these are collectively expressed as a random variable _∈ .

If it is possible to calculate θ (I _local ) for each local region from an RGBD image of the entire container stack at the collection location, then the relationship between the gripping action a targeted at that local region and the gripping weight w can be determined based on the above equation (1), and the robot hand action a (e.g., the insertion depth of the robot hand) required to achieve, for example, a target weight w in that local region can be calculated in reverse.

Taking advantage of this, in this embodiment, θ(I _local ) for each local region is estimated from the RGBD image using machine learning.

It is desirable that θ(I _local ) estimated by machine learning is consistent with the pair of the actual grasping motion a and grasping weight w. Therefore, assuming that _∈ in equation (1) follows a certain probability distribution p( _∈ |I _local ), consider the likelihood as shown in the following equation (2).
Formula (2)

Then, based on the concept of maximum likelihood estimation, machine learning model data is trained so that a parameter θ(I _local ) that maximizes this likelihood can be output.

(Machine learning: local loss function)
Next, the loss function for the local region will be described.

In the learning model of this embodiment, the focus is on grasping a leek with a robotic hand, and the only operational parameter of the robotic hand when grasping the leek in a local area is the insertion depth of the robotic hand.

Regarding the correlation between insertion depth and grip weight, there is a tendency for the deeper the insertion depth, the greater the grip weight, so in this invention, the relationship is approximated by a linear function (one of the main features of the present invention).

Based on this approximation, by using two parameters θ ₀ (I _local ) and θ ₁ (I _local ), the above formula (1) can be expressed as the following formula (3): If the slope (θ ₁ (I _local )) and intercept (θ ₀ (I _local )+ _ε ) of such a linear function formula are identified, then formula (3) can be identified.
Formula (3)

On the other hand, in calculating the likelihood of Equation (2), a specific distribution of _∈ is also required. In this embodiment, a Gaussian distribution is assumed in which the mean is 0 but the standard deviation is σ(I _local ) which varies for each local region. According to this, Equation (2) becomes the following Equation (4).
Formula (4)

To maximize the likelihood calculated by Equation (4), the loss function is the negative log-likelihood, which is then minimized during training of the machine learning model. That is, the loss function is expressed as Equation (5) below.
Formula (5)

From the above, the machine learning model of this embodiment is configured to estimate (output) θ ₀ (I _local ), θ ₁ (I _local ), and σ (I _local ) for each local region. When a pair (set) of actual grip weight w and grip operation parameter a (=insertion depth D) is given, if an estimate can be obtained such that the sum of the loss function calculated by equation (5) for each pair (set) is minimized, the likelihood will be maximized, and a model that well represents the relationship between the actual grip weight w and the grip operation parameter a (=insertion depth D) will be obtained. Furthermore, by estimating not only θ ₀ (I _local ) and θ ₁ (I _local ), but also σ (I _local ), an index of the reliability of the estimation can be obtained.

(Machine learning: model construction)
Next, the configuration of the model will be described.

It is possible to perform a grasping operation in any local area inside the stacked box of green onions, but since a single robot hand only actually grasps one location in each operation, there is no need to estimate the entire area.

For this reason, this embodiment uses a region proposal model that is used in general object recognition, etc. Specifically, the configuration of Faster R-CNN proposed by Ren et al. (Non-Patent Document 2) is used.

Faster R-CNN (Faster Region-based convolutional neural networks) generates region proposals similar to R-CNN (Region-based convolutional neural networks), a deep learning approach.

Faster R-CNN is a model that can output a rectangle enclosing each object in an RGB image, along with an object type label for that rectangle. In other words, the model can be broadly divided into a region proposal unit that proposes a rectangle enclosing an object, and an object identification unit that identifies the object within the proposed region.

In the method proposed in this embodiment, the number of input channels in the first layer is changed from 3 to 4 so that RGBD images that also include distance information from the camera can be input.

The configuration of the region proposal unit was adopted largely as is, but was modified to output the three parameters θ ₀ (I _local ), θ ₁ (I _local ), and σ (I _local ) described in “Loss function for local regions” instead of the object type label for each region, resulting in a model that outputs the relational expression of Equation (3) for each region represented by a rectangle (each local region).

By adopting this model, the image recognition node 72 can determine (propose) multiple candidate local regions and also determine (output) the formula and likelihood of the linear function for each candidate local region.

Here, we will explain the outline of the neural network in the image recognition node 72 of this embodiment, with reference to Figure 8.

In Figure 8, CNN (Convolutional Neural Networks) 76 indicates a convolutional neural network, RPN (Region Proposal Network) 78 indicates a network for extracting candidate image regions where objects may exist from the input image, Roll Pooling 80 indicates a function for extracting partial regions, and FC layers (Fully Connected Layers) 82, 84, 86, and 88 indicate fully connected layers.

As the output of these neural networks, the image recognition node 72 determines (proposes) multiple candidate local regions, and determines (outputs) the equation of the linear function and its likelihood for each candidate local region.

The object identification unit based on Faster R-CNN adopted in this embodiment uses the structure proposed in "Fast R-CNN" shown in Non-Patent Document 1, and the loss function is as shown in the following equation (6) (equation (1) in Non-Patent Document 1).
Formula (6)

Equation (6) is a loss function for a region of interest (RoI), where p is an estimate of the probability distribution of each object class in the region, u is the true object class number, t ^u is an estimate of the coordinates of a bounding box (bbox) that surrounds the object, and v represents the coordinates of the true bbox.

In equation (6) for a given RoI, the first term represents the loss function for the probability distribution of the estimated object class, and the second term represents the loss function for the estimated bbox.

In Fast R-CNN, the first term L _cls is defined as follows using an estimated probability value p _u for the true object class u:
Formula (7)

With reference to these, in this embodiment, the object recognition unit of Faster R-CNN is modified so as to output three parameters θ ₀ (I _local ), θ ₁ (I _local ), and σ(I _local ) instead of the vector p representing the probability distribution of the object class used in equation (7).

Furthermore, instead of equation (7), equation (5) explained in "Local region loss function" is used as the loss function.

In Faster R-CNN, the loss function for the region proposal section includes a term related to whether a certain bbox is an object region to be detected (foreground region) or a background region that does not contain the object to be detected, and it is trained to propose regions containing the object to be detected using this term.

In contrast, in this embodiment, which estimates the relationship (Equation (3)) between the gripping weight of the green onion and the insertion depth D of the robot hand, the gripping operation can be performed in any area inside the stacked box of onions, so it can be said that there is no "background area."

However, to calculate the loss function (equation (5)) that evaluates the parameters of equation (3), information (measured values) on the insertion depth D and grip weight when actually gripping at that location is required. Therefore, for areas where training data on these is unknown, it is not possible to calculate the loss function.

Therefore, in the learning program of this embodiment, such areas where the training data is unknown are treated as background areas, in the sense that although grasping operations are possible, the results are unpredictable, and the loss function of equation (5) is evaluated only for areas where the training data is known.

More specifically, the local area where actual grasping was performed and where training data for insertion depth and grip weight exists is defined as the "training foreground area," and the loss function is calculated for local areas where the IoU (Intersection over Union), an index showing overlap with the training area, is 0.7 or greater (other areas are considered background areas).

If sufficient training data is not available, in order to increase the areas that can be treated as foreground regions as described above when training a learning model, pseudo training data can be assigned to areas where grasping motions are possible but training data is unknown (e.g., approximately 25% of these areas) by selecting, with equal probability, one of the sets of all (a, w) pairs in the training data as shown in Figure 10, as shown in Figure 9.

(Machine learning: training data)
Next, the teacher data will be described.

In this embodiment, the training data is a pair (set) of the "insertion depth" and "grasping weight" of the robot hand measured for each area where an actual grasping operation was performed (this becomes the sample local area).

Specifically, using the system 1, camera 10, and robot hand 8 (soft gripper 14), the following series of operations were performed for 57 pre-set local regions: "divide the RGBD image of the target local region from the RGBD image of the food container, insert the robot hand 8 (soft gripper 14) to an insertion depth D in that local region, grasp and remove the onion, and measure the weight of the grasped and removed onion."

Then, the onions were refilled into the box and the same procedure was repeated.

Through the above procedure, 105 sets of four pieces of information were obtained: "RGBD image of a local area before grasping," "coordinates of the local area," "insertion depth of the robot hand inserted in the local area," and "weight of the leek grasped and removed from the local area."

Figure 11A shows an example of an RGB image obtained by photographing the onions at the collection site with camera 10. Figure 11B shows an example of a D image (distance image, depth image) (colored to clearly indicate depth in analysis) obtained by photographing the onions at the collection site with camera 10. In Figure 11B, the depth values are displayed in color to clearly indicate depth, with lighter grays indicating closer to the camera and darker grays closer to black indicating further away.

Furthermore, the acquired RGBD images may be adjusted by translation, or the brightness and saturation of the RGB images may be changed, thereby increasing the number of RGBD image data points that can be used as training data. For example, 105 sets of acquired RGBD image information examples were divided into 84 examples for training data and 21 examples for evaluation data (a ratio of 80:20). Then, 30 patterns were created for each set (example) of measured data (RGBD image information) using random translation within a range of ±10 pixels in both the vertical and horizontal directions and random brightness changes within a range of ±20% (however, there is a 50% probability of no brightness change). Therefore, 84 x 30 = 2520 examples (2520 sets) of training data and 21 x 30 = 630 examples (630 sets) of training data were obtained.

In this way, in this embodiment, 2,520 sets of teacher data were ultimately prepared as training data, and 630 sets of evaluation data were prepared.

(Machine Learning: Verification)
The neural network model described in "Model Construction" was trained using the training data set. Stochastic gradient descent (SGD) with a learning rate of 1.0 × ¹⁰ was used to minimize the loss function. At each stage of training, the value of the loss function for the evaluation dataset was calculated, and training was terminated at the 39th epoch when it was minimized. The following verification was then performed using the model at this point.

Using this model, estimation was performed on the RGBD image included in the evaluation data, and multiple candidate local regions and parameters θ ₀ (I _local ), θ ₁ (I _local ), and σ(I _local ) for each of the candidate local regions were obtained.

Figure 12 shows an example of candidate local regions proposed for the evaluation data. Multiple virtual square borders (rectangular regions) are displayed as proposed regions on the image of a green onion.

Of these candidate local regions, those that overlap with regions where the actual insertion depth and grip weight are already known were evaluated for which the IoU for both was 0.7 or greater. The results are shown in Figure 13.

In Figure 13, the horizontal axis represents the actual grip weight, and the vertical axis represents the estimated grip weight obtained by substituting the actual insertion depth and estimated _θ0 ( _Ilocal ), _θ1 ( _Ilocal ), and σ( _Ilocal ) into equation (3).

Figure 13 shows that estimates close to the actual measured values are obtained not only for the training data (training data) used in machine learning training, but also for the evaluation data not used in machine learning training.

Furthermore, the root mean square error (RMSE) for the deviation between each measured value and the estimated value for the training data was 0.84 [g], and the root mean square error for the deviation between each measured value and the estimated value for the evaluation data was 1.38 [g]. On the other hand, when all of the training data (see Figure 10) was estimated collectively using a least-squares approximation model with a linear function of grip weight and insertion depth, the root mean square error (RMSE) for the deviation between the measured value and the estimated value for each data was 2.02 [g]. This verified that the learning model of this embodiment improves estimation accuracy.

(Operation and effect of the first embodiment)
According to the system 1 of this embodiment, a candidate local area can be determined from an image of the food container in which the green onions (small ingredients) are collected, and a linear function (approximation) between the insertion depth of the robot hand 8 in the candidate local area and the weight of the green onions being grasped can be estimated (determined) with high accuracy. This makes it easier to respond to changes in the weight of the green onions to be removed.

In particular, according to system 1 of this embodiment, the image recognition node 72 receives machine learning model data from the machine learning node 74, evaluates the similarity (correlation) with multiple sample local regions set (stored) in advance, and determines at least one candidate local region from all local regions. Furthermore, using the machine learning model data (and AI program) constructed by using pairs (combinations) of the "insertion depth" and "grasp weight" of the robot hand measured for each of the multiple sample local regions as training data, the system directly calculates the "slope and intercept of a linear function" corresponding to (linked to) each candidate local region, and also calculates its likelihood. This eliminates the need for system users to be aware of or consider the specifics of the similarity (correlation) between the sample local region and the candidate local region.

Furthermore, according to system 1 of this embodiment, by using a loss function based on the relationship between the gripping motion (insertion depth) and gripping weight to learn the dependency on local regions within a deep learning framework, it is possible to determine candidate local regions and output (estimate) the "slope and intercept of a linear function" corresponding to each candidate local region, resulting in higher estimation accuracy.

Furthermore, according to the system 1 of this embodiment, it is possible to select and use the candidate local region with the highest "likelihood" from among multiple candidate local regions, thereby enabling more accurate control of the weight of the green onions removed.

(Configuration of the second embodiment)
Next, FIG. 14 is a schematic perspective view of a robot hand system 102 including a system 101 according to a second embodiment of the present invention.

The system 101 of the second embodiment is configured to determine the similarity between images using conventional image processing techniques for comparing images, rather than relying on AI machine learning.

In other respects, the configuration of the robot hand system 102 is generally similar to the configuration of the robot hand system 2. Therefore, for the second embodiment, only the differences from the first embodiment will be described, and similar parts will be assigned the same reference numerals and will not be described again.

The system 101 of the second embodiment includes, for example, an overall control node 160 that controls the entire system, and an image processing node 190 that performs processing such as image analysis.

The image processing node 190, together with the overall control node 160, is provided within the control unit of the system 101. The image processing node 190 acquires an RGBD image (which may be an RGB image or a D image) (for example, 100 cm x 55 cm, 1280 pixels x 720 pixels) from the RGBD camera control node 70.

The image processing node 190 compares local regions (e.g., 5 cm x 5 cm, or 90 pixels x 90 pixels) of images acquired from the RGBD camera control node 70 (regions that are offset by a pixel, i.e., may partially overlap) with pre-stored local region images, selects a pair of local region images that are determined to have the highest similarity as a result of the comparison, and determines, from the selected image pair, a local region image that is part of the image acquired from the RGBD camera control node 70 as a candidate local region. It also determines, as the slope and intercept of the linear function corresponding to the candidate local region, information on the slope and intercept of a linear function that indicates the relationship between the insertion depth D of the robot hand 8 into a small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand 8, which information has been linked to a pre-stored local region image from the selected image pair, and sends (outputs) this determined data to the overall control node 160.

The image processing node 190 does not necessarily have to be implemented separately from the overall control node 160, but may be configured as a control unit integrated with the overall control node 160 (for example, an integrated control unit built into a robot). Furthermore, the image processing node 190 can be provided on the cloud, or can be connected to the overall control node 160 via an internet line.

As a preliminary step, the image processing node 190 stores a large number of comparison image data (pre-stored images of specific local areas). The comparison image data is an image of a collection location (e.g., a food container) for small, irregularly shaped ingredients (e.g., green onions) broken down into local areas, and is accompanied by coordinate data indicating their location information.

Then, under the conditions under which each of these comparative image data was captured, a grasping experiment was conducted using the robot hand 8 (soft gripper 14) to grasp a small food ingredient in the corresponding local area, and actual measurements of insertion depth and grasping weight were obtained in advance. By repeating such grasping experiments many times, many data sets (sets) consisting of comparative image data, coordinate data corresponding to the comparative image data, insertion depth data, and grasping weight data are obtained.

The large number of comparison image data can be classified into multiple image groups (e.g., 10 types of image groups) based on, for example, the degree of surface irregularity of the small ingredients (the degree of depth variation in the D images). The image processing node 190 similarly classifies the data sets according to the classification of the image groups, and then, for each image group, calculates, for example, using the least squares method, the slope and intercept of a linear function that indicates the relationship between the insertion depth D of the robot hand 8 into the small ingredient and the weight of the small ingredient that can be grasped and removed by the robot hand 8, and associates this with the image group.

(Operation and effect of the second embodiment)
In substantially the same manner as in the first embodiment, the robot hand system 2 acquires an RGBD image of the accumulation location (food container) of the small ingredients (green onions) using the camera 10.

The image processing node 190 acquires an RGBD image from the RGBD camera control node 70. The image processing node 190 decomposes the RGBD image into local regions.

Next, the image processing node 190 of this embodiment compares each of the local region images, which are part of the RGBD image, with each of the comparison image data (predetermined local region images stored in advance), and selects the pair of local region images that can be determined to have the highest similarity (for example, similarity in the distribution of depth variations in the D image, etc.) as a result of the comparison.

Next, the image processing node 190 of this embodiment determines, as a candidate local region, a local region image that is part of the image acquired from the RGBD camera node 70 from the selected image set, and determines the information on the slope and intercept of a linear function that has been pre-associated with comparison image data (pre-stored local region images) from the selected image set as the slope and intercept of the linear function corresponding to the candidate local region, and transmits (outputs) this determined data to the overall control node 160.

The control information output from the image processing node 190 (e.g., coordinates of determined candidate local regions, information on corresponding linear functions, etc.) may be output to an output unit such as a monitor, either via or without the overall control node 160.

Then, for example, the overall control node 160 determines the insertion depth of the robot hand 8 based on the linear function and the target weight. The overall control node 160 then lowers the soft gripper 14 of the robot hand 8 to the determined insertion depth, and performs the operation of grasping the green onion.

(Other Modifications)
As a further modification, the recognition unit 56 may be provided as an edge computing device on a terminal separate from the robot control unit or on a server provided in a relatively close proximity.

As yet another variation, the recognition unit 56 may be provided on the cloud. In this case, the recognition unit 56 may be connected to multiple robot control units via an internet line, and a common recognition unit may provide image recognition functions to multiple robot control units.

As a further modification, the learning unit 58 may be provided on the cloud and connected to the robot control unit 54 and recognition unit 56 via an internet connection.

1 System 2 Robot hand system 4 Small food ingredient 6 Collection location 8 Robot hand 10 Camera 12 Robot arm 13 Robot arm mounting portion 14 Soft gripper 16 Base member 16a Cross-shaped member 16b Quadrant arc plate 17 Support 20 Scoop element 20a Scoop element outer portion 20b Scoop element inner portion 20c Vertical compressed air supply path 20d Horizontal compressed air supply path 20t Tip portion 31 to 39 Protruding element 41 to 44 Horizontal air chamber 45 to 49 Air chamber 51 Compressed air introduction portion 52 Air supply path 54 Robot control unit 56 Recognition unit 58 Learning unit 60 Overall control node 62 Information recording node 64 Robot hand control node 66 Robot arm control node 68 Image recognition unit management node 70 RGBD camera control node 72 Image recognition node 74 Machine learning node 101 System 102 Robot hand system 105 Actual measurement value 160 Overall control node 190 Image processing node

Claims (15)

A system that outputs control information for controlling a robot hand that can change the insertion depth into a collection location for small, irregularly shaped ingredients,
Acquire an image of the accumulation location of the small food ingredients;
determining at least one candidate local region from the image by evaluating the similarity with a sample local region, and determining the slope and intercept of a linear function associated with each candidate local region that indicates the relationship between the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand, based on information associated with the sample local region that indicates the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand;
A system that links the slope and intercept of a linear function associated with each candidate local region to coordinate data of the candidate local region and outputs the result as control information. The system of claim 1 , wherein the image is a D image. The system of claim 1 , wherein the image is an RGB image. The system of claim 1 , wherein the image is an RGBD image. The system according to claim 1, wherein the determination of the at least one candidate local region and the determination of the slope and intercept of the linear function associated with each candidate local region are performed using a model constructed by machine learning using AI. The system described in claim 5, characterized in that the model is constructed in the machine learning by using multiple pairs of actual measured values of the insertion depth of the robot hand into the small food ingredient and the actual measured values of the weight of the small food ingredient grasped and removed by the robot hand at that time as training data. A system that outputs control information for controlling a robot hand that can change the insertion depth into a collection location for small, irregularly shaped ingredients,
Acquire an image of the accumulation location of the small food ingredients;
determining at least one candidate local region from the image by evaluating the similarity with a sample local region, and determining the slope and intercept of a linear function associated with each candidate local region that indicates the relationship between the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and taken out by the robot hand, and their likelihoods, based on information associated with the sample local region regarding the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and taken out by the robot hand;
A system characterized in that the slope and intercept of a linear function associated with each candidate local region and their likelihood are linked to the coordinate data of each candidate local region and output as control information. When there are a plurality of candidate local regions, the candidate local region with the highest likelihood is determined as the final candidate local region;
The system according to claim 7, wherein the slope and intercept of the linear function associated with the final candidate local region are associated with coordinate data of the final candidate local region and output as control information. The system of claim 7, wherein the image is a D image. The system of claim 7, wherein the image is an RGB image. The system of claim 7, wherein the image is an RGBD image. A robot hand system that takes out small ingredients of irregular shape by grasping them from a collection location of the small ingredients,
A system according to any one of claims 1 to 11;
a camera that captures an image of the collection location of the small food ingredients and transmits the image to the system;
a robot hand that is controlled based on the control information output from the system; and
A robot hand system comprising: A method for outputting control information for controlling a robot hand capable of changing the insertion depth of small ingredients of irregular shape into a collection location, comprising:
acquiring an image of the accumulation location of the small food ingredients;
determining at least one candidate local region from the image by evaluating the similarity with a sample local region, and determining the slope and intercept of a linear function associated with each candidate local region that indicates the relationship between the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and taken out by the robot hand, based on information associated with the sample local region regarding the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and taken out by the robot hand;
a step of linking the slope and intercept of the linear function associated with each candidate local region to coordinate data of each candidate local region and outputting the result as control information;
A method comprising: A method for outputting control information for controlling a robot hand capable of changing the insertion depth of small ingredients of irregular shape into a collection location, comprising:
acquiring an image of the accumulation location of the small food ingredients;
determining at least one candidate local region from the image by evaluating the similarity with a sample local region, and determining the slope and intercept of a linear function associated with each candidate local region that indicates the relationship between the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and taken out by the robot hand, and the likelihood of these, based on information associated with the sample local region regarding the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and taken out by the robot hand;
a step of linking the slope and intercept of the linear function associated with each candidate local region and their likelihoods to the coordinate data of each candidate local region and outputting them as control information;
A method comprising: A program for outputting control information for controlling a robot hand that can change the insertion depth into a collection location for small ingredients of irregular shape,
A program that can implement the method according to claim 13 or 14 when the program is executed by a computer.