JP7728926B2 - system - Google Patents

Description

Article 30, Paragraph 2 of the Patent Act applies. 2021 IEEE International Conference on Robotics and Automation (ICRA) Published on May 2, 2021 https://ras.papercept.net/conferences/conferences/ICRA21/program/ICRA21_ContentListWeb_1.html#tuat20_02

SUMMARY OF THE INVENTION An embodiment of the present invention relates to a system .

Traditionally, food factories that produce bento boxes and other items have required workers to grab a specified target amount of food from a food tray and pack it into a bento box. However, there are many different types of food that can be packed, and the types of food handled change frequently. For these reasons, it is difficult to create specialized machinery for most foods, and the work is carried out manually.

From the perspective of gripping ability and food size, packed foods are classified into foods that are managed by number (3 pieces, 10 pieces, etc.), and small pieces or granular foods that are managed by weight (60g, 100g, etc.). Small pieces or granular foods are difficult to handle, for example, due to their granular shape and irregular shape. Gripping devices for gripping small pieces or granular foods sometimes use specialized tools tailored to the type of food to grip a fixed amount. In this case, a great deal of effort, such as design changes, is required to accommodate a variety of foods and different gripping amounts.

JP 2018-151843 A

The problem that this invention aims to solve is to estimate or grasp the desired amount of grasping of an object to be grasped.

The system according to the embodiment comprises a parallel gripper for gripping food having a first member and a second member opposed to each other and supported by a base portion so as to be able to open and close in a predetermined direction in a horizontal plane, a gripping device equipped with the parallel gripper, and a control means for controlling the operation of the gripping device so as to insert the parallel gripper into the food from above in the vertical direction to grip a portion of the food and release the gripped food at a desired position. In the parallel gripper, the first member and the second member have surface shapes that form a space in which the food is contained when the parallel gripper is closed, the first member and the second member face each other in the closed state and have tip portions that are located vertically below the base portions, the tip portions are formed with straight edges, a first surface shape that intersects with the predetermined direction in the first member is inclined from the base portion to the tip portion toward the second member with respect to the vertical direction, and a second surface shape that intersects with the predetermined direction in the second member is inclined from the base portion to the tip portion toward the first member with respect to the vertical direction, and the first surface shape and the second surface shape move in the predetermined direction in response to the opening and closing of the parallel gripper.

FIG. 1 is a block diagram illustrating an example of the configuration of a grip estimation system including a grip estimation device according to an embodiment. FIG. 2 is a diagram illustrating an example of the appearance of the gripping device according to the embodiment. FIG. 3 is a perspective view of the parallel gripper in an open state according to an embodiment. FIG. 4 is a front view of the parallel gripper in the open state according to the embodiment. FIG. 5 is a perspective view of the parallel gripper in a closed state according to an embodiment. FIG. 6 is a front view of the parallel gripper in the closed state according to the embodiment. FIG. 7 is a perspective view of a parallel gripper according to an embodiment, seen from a different direction. FIG. 8 is a flowchart illustrating an example of a procedure for a gripping process according to the embodiment. FIG. 9 is a diagram illustrating an example of observation data according to the embodiment. FIG. 10 is a diagram showing an example of the estimated amount and uncertainty in a partial region of the photographing range of the tray when the target amount is 60 g according to the embodiment. FIG. 11 is a flowchart illustrating an example of a procedure for an estimation model generation process according to the embodiment. FIG. 12 is a flowchart illustrating an example of a procedure for an evaluation model generation process according to the embodiment. FIG. 13 is a diagram illustrating an example of an outline of an uncertainty assessment model executed as an RND model according to an embodiment. FIG. 14 is a flowchart illustrating an example of a procedure for a model generation process according to the embodiment. FIG. 15 is a diagram illustrating an example of an outline of a gripping amount uncertainty estimation model according to the embodiment. FIG. 16 is a diagram illustrating an example of an outline of a grip amount uncertainty estimation model realized by an ensemble model according to the embodiment. FIG. 17 is a block diagram illustrating an example of the hardware configuration of each device in the embodiment.

The grip estimation device, model generation method, model learning device, and gripper according to the embodiments will be described below with reference to the drawings. Note that the configuration of the embodiment described below is merely an example and is not limited to the contents described below. Furthermore, the actions and effects brought about by the configuration are merely an example and are not limited to the contents described below.

Figure 1 is a block diagram showing an example of the configuration of a gripping estimation system 1 having a gripping estimation device 17. As shown in Figure 1, the gripping estimation system 1 has a gripping device 11, an observation device 13, a weighing device 15, and a gripping estimation device 17. Figure 2 is a diagram showing an example of the appearance of the gripping device 11.

The gripping device 11 includes an end effector 111 that grips the object to be gripped 133, multiple links 113, joints 115 provided between two of the multiple links, and multiple drive devices (motors, etc.) that drive the joints 115 and the end effector 111, respectively. The end effector 111 is realized, for example, by an opening and closing gripper. The opening and closing gripper 111 has, for example, multiple surface shapes that form a space in which food is contained in the closed state. In other words, the gripping device 11 has an opening and closing gripper 111 for gripping food that, in the closed state, forms a space capable of holding the object to be gripped 133 with multiple surface shapes. The surface shapes may be formed of a rigid, non-deformable material, or a deformable material such as an elastic body. For the sake of concreteness, the gripper that realizes the end effector 111 will be described below as a parallel gripper 111, but it may also be a gripper that has one or more rotation axes and opens and closes through the rotational movement of a member. The multiple links 113 and multiple joints 115 give the gripping device 11 multiple degrees of freedom.

Figures 3 to 7 are diagrams showing an example of a parallel gripper 111. Figure 3 shows a perspective view of the parallel gripper 111 in an open state. Figure 4 shows a front view of the parallel gripper 111 in an open state. Figure 5 shows a perspective view of the parallel gripper 111 in a closed state. Figure 6 shows a front view of the parallel gripper 111 in a closed state. As shown by dotted lines in Figures 3 to 6, a force sensor may be installed, for example, at the base portion 121 of the parallel gripper 111. In this case, the force sensor detects the weight of the object 133 gripped by the parallel gripper 111. The force sensor outputs the weight of the object 133 to the gripping estimation device 17. Figure 7 shows a perspective view of the parallel gripper 111 from a different direction. As shown in Figure 7, the parallel gripper 111 is formed of multiple surface shapes 123. As a result, as shown in Figures 5 and 6, the parallel grippers 111 in the closed state form a space 125 in which the food is contained.

The observation device 13 is realized, for example, by various cameras. For example, the camera is realized by an RGB (Red, Green, Blue) camera. The RGB camera is an optical camera that captures images of multiple objects 133 to be grasped placed on the tray 131. In this case, the observation data corresponds to an RGB image. Note that a monochrome camera may be used instead of an RGB camera. Furthermore, the observation device 13 may acquire a type of observation data different from the above observation data. In this case, the observation device 13 is realized by a three-dimensional measurement camera (hereinafter referred to as a D (Depth) camera). Furthermore, the observation device 13 may be realized by an RGB camera and a D camera (hereinafter referred to as an RGB-D camera). For concrete explanation, the observation device 13 is assumed to be realized by an RGB-D camera. The capture range of the RGB-D camera is a range that includes the entire area of the tray 131 on which the objects 133 to be grasped are placed.

The weighing device 15 is, for example, placed on the underside of the tray 131. The weighing device 15 corresponds to, for example, a scale that measures the total weight of the tray 131 and the grasped object 133 placed on the tray 131. The weighing device 15 outputs measurement data associated with weighing the tray 131 to the grasp estimation device 17. The measurement data output from the weighing device 15 is used, for example, in generating (learning) a grasp amount estimation model and an uncertainty assessment model described below. Therefore, once the generation of the grasp amount estimation model and the uncertainty assessment model is complete, the weighing device 15 does not need to use the grasp estimation system 1. Furthermore, if a force sensor is provided on the parallel gripper 111, the weighing device 15 does not need to be placed on the underside of the tray 131.

The grip estimation device 17 has a controller 171. As shown in FIG. 1, the controller 171 has an acquisition unit 173, an estimation unit 175, an evaluation unit 177, a selection unit 179, and a control unit 181.

The acquisition unit 173 acquires observation data of the object to be grasped 133. As described above, the observation data includes image data of the object to be grasped 133. The observation data also includes distance data from the acquisition position of the observation data to the surface of the object to be grasped 133. In other words, the distance data is data indicating the distance from the installation position of the D camera, which is the acquisition position of the observation data, to the surface of the object to be grasped 133. The object to be grasped 133 is, for example, various beans such as coffee beans or granular food such as rice. Note that the acquisition unit 173 may acquire a type of observation data different from the observation data. Specifically, the acquisition unit 173 acquires an RGB image and a D image. More specifically, the acquisition unit 173 acquires observation data for each of multiple partial regions in the distribution region of the object to be grasped 133. The distribution region of the object to be grasped 133 corresponds, for example, to the region in the tray 131 where the object to be grasped can be placed. Each of the multiple partial regions is an area of a predetermined size, and corresponds to an area (cut area: patch) cropped from the RGB image and the D image. The partial regions are sometimes called gripping points. When generating each model described below, the acquisition unit 173 acquires measurement data from the weighing device 15, or acquires the weight of the gripping object 133 gripped by the end effector 111 from a force sensor.

The estimation unit 175 estimates the gripping amount of the gripping object 133 gripped by the gripping device 11 based on the observation data. The estimated gripping amount is called the estimated amount. Specifically, the estimation unit 175 estimates the gripping amount in each of the plurality of partial regions in the observation data. The estimation unit 175, for example, inputs the observation data into a trained gripping amount estimation model that has been trained to input the observation data of the gripping object 133 and output an estimated amount, and estimates the gripping amount. More specifically, the estimation unit 175 inputs each of the plurality of partial regions in the observation data into the trained gripping amount estimation model, and estimates the gripping amount in each of the plurality of partial regions.

The grip amount estimation model is generated by training the first learning model using, for example, the actual measured value of the grip amount of the gripping object 133 gripped by the gripping device 11 as training data for the first learning model, and observation data of the gripping object 133 before being gripped by the gripping device 11 as input data for the first learning model. The procedure for generating the grip amount estimation model by training the first learning model will be described later.

The evaluation unit 177 evaluates, based on the observation data, the uncertainty that the gripping device 11 will grasp only the grip amount estimated by the trained grip amount estimation model. The uncertainty index (uncertainty index) may be, for example, an uncertainty degree, which is a degree (or percentage) indicating the uncertainty that the gripping device 11 will grasp only the estimated grip amount. In other words, the uncertainty index is an index for evaluating whether the actual grip amount is close to the estimated amount. Note that, instead of the uncertainty degree, the uncertainty index may be a degree (or percentage) indicating the certainty that the gripping device 11 will grasp only the estimated grip amount, i.e., the reliability that the gripping device 11 will grasp the estimated grip amount.

To simplify the explanation below, uncertainty may be interpreted as the degree of uncertainty (degree of uncertainty) or an uncertainty index, as appropriate. Furthermore, the evaluation unit 177 may evaluate the uncertainty of only the estimated quantity being grasped by the grasping device 11 based on a type of observation data different from the observed data. Specifically, the evaluation unit 177 evaluates the uncertainty in each of a plurality of partial regions based on the observed data. More specifically, the evaluation unit 177 inputs the observed data into a trained uncertainty assessment model that has been trained to input the observed data of the grasp target 133 and output an uncertainty index, and evaluates the uncertainty.

The uncertainty assessment model is generated by self-supervised learning using observation data of the grasped object 133. This self-supervised learning inputs the observation data into a learning model, outputs data based on the model's internal parameters (e.g., weights and biases in a neural network), generates training data (correct answer data) from the observation data, and updates the model's internal parameters based on the output data and training data. The procedure for generating an uncertainty assessment model will be explained later.

The selection unit 179 selects from the plurality of partial regions an area where the object to be grasped is to be grasped by the grasping device 11, based on the uncertainty evaluated for each of the plurality of partial regions. For example, the selection unit 179 selects from the plurality of partial regions a position to be grasped by the grasping device 11 (hereinafter referred to as a grasping position) based on the estimated amount and uncertainty corresponding to each of the plurality of partial regions and predetermined conditions. The predetermined condition is, for example, a predetermined weight range (hereinafter referred to as a predetermined range) relative to a target grasping amount (hereinafter referred to as a target amount) set by the user. The predetermined range is, for example, a range from +0.5 g to -0.5 g. The predetermined condition (e.g., the predetermined range) is stored in the memory of the grasping estimation device 17. The predetermined condition may include a predetermined uncertainty range (e.g., an uncertainty of 0.1 or less).

Specifically, the selection unit 179 selects, from among the multiple estimated quantities corresponding to the multiple partial regions, multiple estimated quantities that fall within a predetermined range, using the target quantity as a reference. Next, the selection unit 179 identifies the estimated quantity with the smallest degree of uncertainty, i.e., the lowest uncertainty, from among the multiple uncertainties corresponding to the selected multiple estimated quantities. The selection unit 179 selects the partial region for the identified estimated quantity with the lowest uncertainty as the grasping position to be grasped by the grasping device 11. Note that when reliability is used as the uncertainty index, the selection unit 179 identifies, from among the multiple reliabilities corresponding to the selected multiple estimated quantities, the partial region for the estimated quantity with the highest degree of reliability, i.e., the lowest uncertainty.

Note that the method for selecting a grasping position by the selection unit 179 is not limited to the above. The selection unit 179 may select a grasping position based on the target amount, the estimated amount, and the uncertainty of the estimated amount. For example, the selection unit 179 may select a grasping position taking into consideration the balance between the difference between the target amount and the estimated amount and the uncertainty. The selection unit 179 may compare a partial region where the difference between the target amount and the estimated amount is equal to or less than a threshold but the uncertainty exceeds the threshold with a partial region where the difference between the target amount and the estimated amount exceeds the threshold but the uncertainty is equal to or less than the threshold, and select the latter. In this case, when the estimated amount is less than the target amount, the currently grasped object may be released at the target position, and then a second grasping operation may be performed with the difference between the estimated amount and the target amount as the new target amount. Furthermore, when the estimated amount exceeds the target amount, the excess amount of the currently grasped object may be dropped and then released at the target position (described below). In this way, variations in grasping strategies may be prepared that take into account the evaluated grasping uncertainty.

The control unit 181 controls the operation of the gripping device 11 (the gripping operation, movement operation, and release operation of the gripping object 133). Specifically, the control unit 181 controls the gripping device 11 to grip the gripping object 133 at the selected gripping position. As a result, the gripping device 11 grips the gripping object 133 at the gripping position. The control unit 181 controls the gripping device 11 to move the gripped gripping object 133 to a predetermined target position and release the gripping object 133 at this target position. Known control is used appropriately to control the movement operation and release operation, so a description thereof will be omitted.

The configuration of the grip estimation system 1 has been described above. Below, we will explain the processing related to gripping movements (hereinafter referred to as grip processing) performed by the grip estimation system 1. Figure 8 is a flowchart showing an example of the grip processing procedure.

(Gripping process)
(Step S801)
The acquisition unit 173 acquires observation data from the observation device 13 for each of a plurality of partial regions of a predetermined size related to the gripping target 133. The predetermined size is stored in advance in the memory of the gripping estimation device 17 as a size that can be gripped by the parallel gripper 111. FIG. 9 is a diagram showing an example of the observation data. As shown in FIG. 9, the acquisition unit 173 acquires, for example, an RGB image and a D image of the granular food item, which is the gripping target 133, placed on the tray 131, for each of the plurality of partial regions.

(Step S802)
The estimation unit 175 inputs the observation data for each partial region into the grip amount estimation model and estimates the grip amount of the gripping target 133 grasped by the gripping device 11. As a result, the estimation unit 175 outputs a plurality of estimated amounts corresponding to a plurality of partial regions.

(Step S803)
The evaluation unit 177 inputs the observation data for each partial region into an uncertainty assessment model and outputs an uncertainty index from the uncertainty assessment model. That is, the evaluation unit 177 evaluates the uncertainty of whether the grasp target 133 will be grasped by the estimated amount. In this way, the evaluation unit 177 evaluates the uncertainty of the estimated amount for each partial region.

(Step S804)
The selection unit 179 identifies a region where the estimated gripping amount is within a predetermined range (e.g., ±0.5 g) from the target amount. FIG. 10 is a diagram showing an example of the estimated amount and uncertainty in a portion of the imaging range of the tray 131 when the target amount is 60 g. The dotted circles in FIG. 10 indicate regions for multiple estimated amounts. The solid squares in FIG. 10 indicate multiple regions identified by the selection unit 804 and the uncertainty in those regions. Note that if there is no region with an estimated amount within a predetermined range from the target amount, the control unit 181 may notify an error, or the control unit 181 may control the gripping device 111 to level the gripping target 133 placed on the tray 131 with the gripping device 111 before returning the process to step S801.

(Step S805)
The selection unit 179 selects the region with the lowest uncertainty (i.e., the lowest uncertainty or the highest reliability) from the identified regions. In FIG. 10 , the region MR with the lowest uncertainty corresponds to a region with an uncertainty of 0.1. Note that if the predetermined condition includes a predetermined uncertainty range (e.g., an uncertainty of 0.1 or less), a region with an uncertainty within this uncertainty range is selected. If there is no region with uncertainty within this uncertainty range, the control unit 181 may notify an error via the output device, or the control unit 181 may control the gripping device 111 to level the gripping target 133 placed on the tray 131 with the gripping device 111 and then return the process to step S801.

(Step S806)
The control unit 181 controls the gripping device 11 to grip the gripping target 133 from the selected region, thereby completing the gripping process.

The grasping estimation device 17 according to the embodiment described above acquires observation data of the grasping object 133, which may be, for example, a granular food item, and estimates the grasping amount of the grasping object 133 grasped by the grasping device 11 based on the acquired observation data. Specifically, the grasping estimation device 17 inputs the acquired observation data into a trained grasping amount estimation model that has been trained to estimate the grasping amount using the observation data of the grasping object 133 as input, and estimates the grasping amount. In this way, the grasping estimation device 17 can estimate the grasping amount desired by the user from the observation data.

The grip estimation device 17 also evaluates the uncertainty of the estimated grip amount being grasped by the grip device 11 based on the observation data. The grip estimation device 17 also acquires a type of observation data different from the observation data, and evaluates the uncertainty of the estimated grip amount being grasped by the grip device 11 based on the acquired type of observation data. For example, the grip estimation device 17 inputs the observation data of the grip target 133 into a trained uncertainty assessment model that has been trained to output an uncertainty index using the input observation data, and evaluates the uncertainty. The observation data in the grip estimation device 17 includes image data of the grip target 133 or distance data from the acquisition position of the observation data to the surface of the grip target 133. Specifically, the grip estimation device 17 according to the embodiment acquires observation data for each of multiple partial regions in the distribution region of the grip target 133, estimates the grip amount for each of the multiple partial regions, evaluates the uncertainty for each of the multiple partial regions, and selects a region to be grasped by the grip device 11 from the multiple partial regions based on the grip amount and uncertainty corresponding to each of the multiple partial regions and predetermined conditions. For these reasons, the grip estimation device 17 can grasp the desired grip amount for the gripping object 133 in an area of the estimated grip amount where the degree of uncertainty is low.

The gripping estimation device 17 in this embodiment also has a parallel gripper 111 that, in a closed state, forms a space capable of holding the gripping object 133 using multiple surface shapes 123. For example, the parallel gripper 111 for gripping food according to this embodiment has multiple surface shapes 123 that form a space in which the food is contained when closed. The parallel gripper 111 for gripping food according to this embodiment is also made of a non-deformable member, and is realized, for example, by a parallel gripper. As a result, the parallel gripper 111 for gripping food according to this embodiment can appropriately grip a variety of irregularly shaped and deforming foods. Therefore, the parallel gripper 111 does not require dedicated tools or end effectors for gripping fixed amounts, which reduces the significant effort required for design changes, etc.

From the above, the grip estimation device 17 and parallel gripper 111 for gripping food in this embodiment can estimate or grasp the desired grip amount for the gripping object 133.

The process for generating a grip amount estimation model and a process for generating an uncertainty assessment model are described below. The processes for generating a grip amount estimation model and a process for generating an uncertainty assessment model each correspond to a learning process. In this case, the grip estimation system 1 or grip estimation device 17 shown in FIG. 1 functions as a model learning device. First, the process for generating a grip amount estimation model is described, and then the process for generating an uncertainty assessment model is described. FIG. 11 is a flowchart showing an example of the steps in the process for generating a grip amount estimation model (hereinafter referred to as the estimation model generation process).

(Estimation model generation process)
(Step S111)
The acquisition unit 173 acquires observation data for a partial region of a predetermined size related to the grasp target 133 from the observation device 13. The acquisition unit 173 outputs the position of the partial region to the control unit 181. The acquisition unit 173 also stores the position of the partial region in memory.

(Step S112)
The estimation unit 175 inputs the observation data relating to the partial region into the first learning model, and outputs (estimates) the gripping amount of the gripping object 133 grasped by the gripping device 11 as a first output value.

(Step S113)
The control unit 181 controls the gripping device 11 so as to grip the gripping target 133 in the partial region. The gripping device 11 grips the gripping target 133 in the partial region under the control of the control unit 181.

(Step S114)
The acquisition unit 173 acquires an actual measurement value of the gripping amount by the gripping device 11, i.e., the actual gripping amount, based on the output from the weighing device 15 or the force sensor. When calculating the actual measurement value based on the measurement data output from the weighing device 15, the acquisition unit 173 acquires the actual measurement value, for example, by calculating the difference between the measurement data before and after the gripping device 11 grips the gripping target 133. Note that if a force sensor is mounted on the parallel gripper 111, the acquisition unit 173 acquires the actual measurement value by converting the output from the force sensor into weight.

(Step S115)
The estimation unit 175 calculates a loss based on the difference between the first output value (estimate) and the actual measurement value. For example, the estimation unit 175 calculates the loss (squared error) by applying the difference to a loss function. Next, the estimation unit 175 updates the parameters in the first learning model so as to reduce the loss by using a known method such as backpropagation or stochastic gradient descent.

(Step S116)
If the number of parameter updates is the predetermined number (Yes in step S116), the process of step S118 is executed. If the number of parameter updates is not the predetermined number (No in step S116), the process of step S117 is executed. The predetermined number is set in advance by a user or the like. Note that the process in this step is not limited to comparing the number of parameter updates with the predetermined number. For example, the processes of steps S111 to S116 may be repeated until learning for the first learning model converges. Convergence of learning for the first learning model refers, for example, to when the difference between the estimated amount and the actual measured value is equal to or less than a predetermined value. The predetermined number or the predetermined value is stored in advance in a memory in the grip estimation device 17.

(Step S117)
The acquisition unit 173 sets another partial area in the observation data that is different from the partial area set in step S111. The acquisition unit 173 outputs the position of the other partial area to the control unit 181.

(Step S118)
The estimation unit 175 sets the first learning model as the grip amount estimation model. The estimation unit 175 stores the grip amount estimation model in the memory of the grip estimation device 17. This completes the estimation model generation process.

According to the estimation model generation process according to the embodiment described above, the grip amount estimation model used in the grip estimation device 17 is generated by learning the first learning model using the actual measured value of the grip amount of the gripping object 133 gripped by the gripping device 11 as training data for the first learning model and observation data before the gripping object 133 is gripped by the gripping device 11 as input data for the first learning model.

The process for generating an uncertainty assessment model is described below. Figure 12 is a flowchart showing an example of the steps in the process for generating an uncertainty assessment model (hereinafter referred to as the assessment model generation process).

(Evaluation model generation process)
(Step S121)
The acquisition unit 173 acquires observation data for a partial region of a predetermined size related to the grasp target 133 from the observation device 13. The acquisition unit 173 also stores the position of the partial region in memory.

(Step S122)
The evaluation unit 177 inputs the observation data for the partial region into the second learning model and outputs a second output value. As will be understood from the following explanation, the evaluation unit 177 estimates, as the second output value output from the second learning model, data (output value) output by a reference model (a fixed weight model or target net described below) other than the second learning model based on the observation data.

(Step S123)
The evaluation unit 177 inputs the observation data for the partial region into the fixed weight model and outputs a third output value. The fixed weight model corresponds to, for example, a model with fixed weights in a neural network model. The fixed weight model is a reference neural network model and is called a target net. The fixed weight model and the second learning model are different neural network models, differing in hyperparameters, such as the number of intermediate layers. Furthermore, the fixed weight model and the second learning model are preferably models with different neural network structures, such as an image net. The fixed weight model may be configured so that its statistical properties are similar to those of the observed data, even if the values themselves differ from those obtained from the actual observed data, by artificially adding minute noise within a predetermined range to the observed data input to the fixed weight model or the output obtained from the fixed weight model, or by adjusting the statistical quantities, such as the mean and variance of the values output from the fixed weight model, so that they match the observed data.

(Step S124)
The evaluation unit 177 calculates the difference between the second output value and the third output value. Then, the evaluation unit 177 calculates the loss (squared error) based on the calculated difference and the loss function. This difference corresponds to the uncertainty index output by the uncertainty evaluation model.

(Step S125)
The evaluation unit 177 updates the parameters in the second learning model so as to reduce the loss by using a known method such as backpropagation or stochastic gradient descent.

(Step S126)
If the number of parameter updates is the predetermined number (Yes in step S126), the process of step S128 is executed. In this case, the second learning model corresponds to a learned model, and is called, for example, a predictor net. Learning for the second learning model corresponds to self-supervised learning. If the number of parameter updates is not the predetermined number (No in step S126), the process of step S127 is executed. Note that the process in this step is not limited to comparing the number of parameter updates with the predetermined number. For example, the processes of steps S121 to S126 may be repeated until learning for the first learning model converges. Convergence of learning for the second learning model refers, for example, to a case where the difference between the second output value and the third output value is equal to or less than a predetermined value.

(Step S127)
The acquisition unit 173 sets another partial area in the observation data that is different from the partial area set in step S121. The acquisition unit 173 outputs the position of the other partial area to the control unit 181.

(Step S128)
The evaluation unit 177 collectively sets the predictor net and the target net as an uncertainty evaluation model. In this case, the uncertainty evaluation model corresponds to an RND (Random Network Distillation) model. In this case, the uncertainty evaluation model corresponds to a model that evaluates whether input observation data is close to the data used during learning, i.e., the degree of similarity. The evaluation unit 177 stores the uncertainty evaluation model in the memory of the grip estimation device 17. In this case, the output from the uncertainty evaluation model is a value indicating the difference between the output from the fixed weight model and the output from the trained uncertainty evaluation model, and is expressed, for example, in the range from 0 (no difference) to 1 (maximum difference). When the output from the uncertainty evaluation model is 0, the degree of uncertainty corresponds to 0 and the reliability is maximum. On the other hand, when the output from the uncertainty evaluation model is 1, the degree of uncertainty corresponds to maximum and the reliability corresponds to 0. This completes the evaluation model generation process.

Figure 13 shows an example of an outline of an uncertainty assessment model implemented as an RND model. As shown in Figure 13, in the RND model, an RGB image (cropped RGB) of a subregion and a D image (cropped depth) of the subregion are input to a predictor net and a target net, respectively. The RND model outputs the difference between the output from the predictor net and the output from the target net as an uncertainty index (estimation error).

According to the evaluation model generation process according to the embodiment described above, the uncertainty evaluation model used in the grip estimation device 17 is generated by training the second learning model so that, when observation data is input, the second learning model outputs data that is the same as the data output by the target net based on the observation data.

Although the above description illustrates an example in which the estimation model generation process and the evaluation model generation process are executed separately, the grip amount estimation model and uncertainty evaluation model can also be generated simultaneously. In this case, the grip amount estimation model and uncertainty evaluation model are collectively referred to as the grip amount uncertainty estimation model (see Figure 15). The process for generating the grip amount uncertainty estimation model (hereinafter referred to as the model generation process) is described below. Figure 14 is a flowchart showing an example of the model generation process procedures. The processes of steps S141, S142, S144, S145, S148, and S149 in Figure 14 are similar to the processes of steps S111, S112, S113, S114, S116, and S117 in Figure 11, respectively, and therefore will not be described again.

(Model generation process)
(Step S143)
The evaluation unit 177 inputs observation data related to the partial region into the third learning model and outputs an error amount of the estimated amount of the grasp target 133 grasped by the grasping device 11. The error amount corresponds to the difference between the actual measurement value of the grasp amount of the grasp target 133 actually grasped by the grasping device 11 in the partial region and the estimated amount output by the grasp amount estimation model. In other words, the third learning model estimates the error of the estimated amount of the grasp target 133 grasped from the partial region (grasp amount estimation error) based on the observation data related to the partial region. This error (error amount) corresponds to the uncertainty (uncertainty index).

(Step S146)
The estimation unit 175 calculates the difference (hereinafter referred to as the first difference) between the first output value (estimated amount) and the actual measurement value. The evaluation unit 177 calculates the difference (hereinafter referred to as the second difference) between the error amount and the first difference.

(Step S147)
The estimation unit 175 calculates a first loss (squared error) by, for example, applying the first difference to a loss function. Then, the estimation unit 175 updates parameters in the first learning model so as to reduce the first loss by a known method such as backpropagation or stochastic gradient descent. The evaluation unit 177 calculates a second loss (squared error) by, for example, applying the second difference to the loss function. Then, the evaluation unit 177 updates parameters in the third learning model so as to reduce the second loss by a known method such as backpropagation or stochastic gradient descent.

(Step S150)
The estimation unit 175 sets the trained first learning model as the grip amount estimation model. The evaluation unit 177 sets the trained third learning model as the uncertainty evaluation model. At this time, the uncertainty evaluation model corresponds to an error estimation model that estimates the estimation error (uncertainty) of the grip amount estimated by the grip amount estimation model. Specifically, the estimation unit 175 and the evaluation unit 177 combine the trained first learning model and the trained third learning model and set them as the grip amount uncertainty estimation model. The estimation unit 175 and the evaluation unit 177 store the set grip amount uncertainty estimation model in the memory of the grip estimation device 17. At this time, the processes of steps S802 and S803 in the grip processing shown in FIG. 8 can be summarized as follows. That is, in the processes in steps S802 and S803, the observation data for each partial region is input to the grip amount uncertainty estimation model, and an estimated amount and uncertainty (uncertainty index) are output. The learning of the third learning model in the model generation process corresponds to self-supervised learning.

Figure 15 shows an example of an overview of the grip amount uncertainty estimation model. As shown in Figure 15, the grip amount uncertainty estimation model receives an RGB image (cropped RGB) of a partial region and a D image (cropped depth) of the partial region. The grip amount uncertainty estimation model outputs an estimated amount and a degree of uncertainty (uncertainty index).

The grip amount uncertainty estimation model described above is a combination of separate models, namely the first learning model (grip amount estimation model) and the third learning model (uncertainty assessment model). This provides the following advantages: The grip amount estimation model and the uncertainty assessment model can be trained independently, allowing for separate parameter tuning for each model. Furthermore, the combination of the grip amount estimation model and the uncertainty assessment model can be freely selected as appropriate, and the RND model may be combined in place of the error estimation model described above, or both models may be combined.

However, on the other hand, the grip amount uncertainty estimation model can also be configured as a single model. That is, a single model (preferably a single neural network model) that receives observation data related to a subregion and outputs an estimate of the grip amount and an error of that estimate can be trained, and this trained model can be set as the grip amount uncertainty estimation model. Benefits of training with a single model include a shorter training time and a smaller memory capacity for storing the model. This model is trained, for example, using the following procedure. The estimation unit 175 and the evaluation unit 177 input the observation data into the model and output an estimate of the grip amount and an error of that estimate. The estimation unit 175 and the evaluation unit 177 calculate a first loss related to the output estimate and a second loss related to the output error, and then add the calculated first loss and second loss to calculate a total loss. The estimation unit 175 and the evaluation unit 177 update the parameters of the model to reduce the total loss using known methods such as backpropagation or stochastic gradient descent. The model trained in this way is obtained as the grasp amount uncertainty estimation model.

In the above explanation, the uncertainty assessment model has been described as an RND model or an error estimation model. However, the uncertainty assessment model is not limited to an RND model or an error estimation model and can be realized using other methods. For example, the uncertainty assessment model may be realized by an ensemble using multiple grip amount estimation models. An ensemble uncertainty assessment model (hereinafter referred to as an ensemble model) uses, for example, multiple grip amount estimation models that have different neural network model structures or different training data, and uses the variation (variance) of the output (estimated amount) from each grip amount estimation model as an uncertainty index (e.g., degree of uncertainty).

Figure 16 is a diagram illustrating an example of an outline of a grip amount uncertainty estimation model realized by an ensemble model. As shown in Figure 16, the ensemble model has multiple grip amount estimation models. The evaluation unit 177 inputs an RGB image (cropped RGB) of a partial region and a D image (cropped depth) of the partial region to each of the multiple grip amount estimation models. The evaluation unit 177 calculates an uncertainty index based on the outputs (estimates) from the multiple grip amount estimation models in the ensemble model. The calculated uncertainty index corresponds to, for example, a standard deviation or variance normalized based on the multiple estimates. As a result, the evaluation unit 177 outputs the uncertainty index. Note that the estimation unit 175 may calculate the average (average estimate) of the multiple estimates output from the multiple grip amount estimation models in the ensemble model, and output this as the estimate in step S802.

According to the ensemble model generation process according to the embodiment described above, the ensemble model used in the grip estimation device 17 uses multiple trained grip amount estimation models, and can use the variability in the output (estimated amount) from each grip amount estimation model as an uncertainty index.

According to the model learning device of this embodiment, observation data of the grasp target 133 is acquired, and the actual measured value of the grasp amount of the grasp target 133 grasped by the grasp device 11 is used as training data for the first learning model. The observation data is used as input data for the first learning model to train the first learning model, generating a grasp amount estimation model that inputs the observation data and outputs a grasp amount (estimated amount). The first model generation unit that generates the grasp amount estimation model is realized, for example, by the estimation unit 175. Furthermore, according to this learning device, a learning model that receives the observation data and outputs an uncertainty index is trained using self-supervised learning, thereby generating an uncertainty assessment model that outputs an uncertainty index indicating that only the estimated grasp amount (estimated amount) is grasped from the input observation data. Examples of training data for this self-supervised learning are "the difference between the grasp amount (estimated amount) output when the observation data is input to the grasp amount estimation model and the actual measured value" or "the difference between the first output value output when the observation data is input to a specified model and the second output value output when the observation data is input to the learning model." The second model generation unit that generates the uncertainty assessment model is realized by, for example, the evaluation unit 177. Either the first model generation unit or the second model generation unit, or both together, may be referred to as the model generation unit. The first model generation unit and the second model generation unit may be provided separately in the grip estimation device 17. Furthermore, with regard to model learning, the model learning device does not necessarily need to generate both the grip amount estimation model and the uncertainty assessment model, and may generate either the required model.

As described above, the grasping estimation device 17 according to this embodiment can deal with the imperfection of the estimated grasping amount by using an uncertainty assessment model to model the uncertainty of grasping only the estimated amount using self-supervised learning, and can grasp the grasping object 133 at the most reliable grasping position among those closest to the grasping amount (target amount) specified by the user, even if the output results of the grasping amount estimation model become unreliable for a granular food object 133 that is easily deformed, has significant interactions between individual particles, and is difficult to simulate or model accurately, for example, due to factors such as food deterioration, damage caused by grasping, and moisture, making it difficult to collect training data by repeatedly using the same food, i.e., for collecting sufficient training data.

In other words, the grip estimation device 17 according to this embodiment can select the most reliable grip position from the output results of the grip amount estimation model based on the output from the uncertainty assessment model. Therefore, even if learning data is collected in a shorter time than conventional methods and the grip amount estimation model, uncertainty assessment model, and grip amount uncertainty estimation model are trained in a shorter time, it is possible to accurately grasp the target amount of the grip object 133.

When the technical concepts of the estimation model generation process, uncertainty model generation process, and model generation process are realized by a model generation method (which may also be called a model manufacturing method), the model generation method generates a grip amount estimation model and an uncertainty assessment model using processing procedures similar to those of the embodiments. Furthermore, the model generation method is executed by a model learning device.

As described above, this embodiment attempts to grasp a target amount of an object by estimating the grasp amount and evaluating the uncertainty of that estimation before using the gripper to grasp the object. However, there are cases where the actual amount grasped by the gripper exceeds the target amount. To prepare for such cases, the grasp estimation device 17 according to this embodiment may achieve grasping the target amount by dropping the excess amount of the object that was actually grasped from the gripper. As an example of a method for dropping the excess amount, a method of opening a closed gripper by a gap width y (e.g., 0.5 mm) for an opening time x (e.g., 0.5 seconds) will be described.

(Method 1)
The estimation unit 175 learns a learning model (preferably a neural network model) that inputs the gripper's actual gripping amount p, gripper release time x, and gap width y and outputs the amount z of the gripped object that falls through the gap, and stores the learning model in memory in the gripping estimation device 17. This learning involves inputting the inputs p, x, and y to the learning model to obtain the output z, calculating a loss based on the output z and the amount of the gripped object that actually falls from the gripper (fall amount) z' when the gap width y is opened for the release time x when the amount actually gripped by the gripper is gripping amount p, and updating the parameters of the learning model in a direction that reduces the loss using a known method such as backpropagation or stochastic gradient descent based on the loss. The estimation unit 175 then uses this trained model stored in memory to calculate gripper control amounts x and y for dropping the surplus amount from the gripper and setting the gripping amount to the target amount. Specifically, the estimation unit 175 (1) sets values for the candidate control amounts x' and y', (2) inputs x' and y' and the current gripping amount p into the trained model to estimate the drop amount z, and determines whether this drop amount z is close to the difference between the current gripping amount p and the target amount (i.e., whether this drop amount z is within a predetermined range (e.g., within ±0.5 g) of this difference). (3) If they are close, the estimation unit 175 passes these x' and y' to the control unit 181 as the control amounts of the gripper. (4) If they are not close, the estimation unit 175 sets another value to one or both of x' and y' and returns to process (2). The control unit 181 controls the release of the gripper according to the passed control amount, thereby dropping the excess amount of the gripped object from the gripper and achieving the target amount of gripping.

(Method 2)
In Method 1, the output z of the learning model was the amount of fall, but it may also be the remaining amount of the object to be grasped remaining in the gripper. Even in this case, the model can be learned using the same framework as Method 1. In this case, the estimation unit 175 may replace the amount of fall in Method 1 with the remaining amount z, and determine whether the remaining amount z is close to the target amount in the determination of (2).

(Method 3)
While a trained model is used to estimate the drop amount in Method 1, a table having items such as grip amount p, release time x, gap width y, and drop amount z may also be used. In this case, the values of each pair of p, x, y, and z in the table may be experimentally determined in advance or may be determined by interpolating between experimentally determined values. When dropping the excess amount, the estimation unit 175 may estimate the drop amount z by searching the table using x' and y' and the current grip amount p as keys in estimating the drop amount z in Method 1 (2). Alternatively, the estimation unit 175 may consider the difference between the current grip amount p and the target amount as the drop amount z, search the table using the current grip amount p and drop amount z as keys to identify the control amounts x and y, and pass these x and y to the control unit 181 as the gripper control amounts.

(Method 4)
Instead of the drop amount z in the table of Method 3, the remaining amount z remaining in the gripper may be used as an item. In this case, a table can be prepared as in Method 3. In this case, the estimation unit 175 may replace the estimation of the drop amount z described in Method 3 with an estimation of the remaining amount z, and determine whether the estimated remaining amount z is close to the target amount in the determination. Alternatively, the estimation unit 175 may regard the target amount as the remaining amount z, search the table using the current gripping amount p and the remaining amount z as keys to identify the control amounts x and y, and pass these x and y to the control unit 181 as the control amounts of the gripper.

For methods 1 to 4 above, the uncertainty of the drop amount z or remaining amount z may also be evaluated. In this case, when selecting the control variables x and y to obtain the desired drop amount z or remaining amount z, it is sufficient to select the control variables x and y with small uncertainty.

Some or all of the devices in the above-described embodiments (such as the grip estimation device 17 and/or controller 171) may be configured as hardware, or may be configured as software (program) information processing executed by a CPU (Central Processing Unit) or GPU (Graphics Processing Unit). If configured as software information processing, software that realizes at least some of the functions of the devices in the above-described embodiments (such as the grip estimation device 17 and/or controller 171) may be stored on a non-transitory storage medium (non-transitory computer-readable medium) such as a flexible disk, CD-ROM (Compact Disc-Read Only Memory), or USB (Universal Serial Bus) memory, and the software information processing may be executed by loading the software into a computer. The software may also be downloaded via a communications network. Furthermore, information processing may be performed by hardware by implementing the software in a circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

There are no restrictions on the type of storage medium that stores the software. The storage medium is not limited to removable media such as magnetic disks or optical disks, but can also be fixed storage media such as hard disks or memory. Furthermore, the storage medium may be installed inside the computer or external to the computer.

Figure 17 is a block diagram showing an example of the hardware configuration of each device (such as the grip estimation device 17 and/or the controller 171) in the above-described embodiment. Each device may be realized as a computer 81 including, for example, a processor 811, a main storage device 812 (memory), an auxiliary storage device 813 (memory), a network interface 814, and a device interface 815, all of which are connected via a bus 816.

The computer 81 in FIG. 17 includes one of each component, but may include multiple of the same component. Also, while FIG. 17 shows a single computer 81, the software may be installed on multiple computers, with each of the multiple computers executing the same or different parts of the software. In this case, a form of distributed computing may be used, in which each computer communicates via a network interface 814 or the like to execute processing. In other words, each device in the above-described embodiment (such as the grip estimation device 17 and/or controller 171) may be configured as a system in which one or more computers execute commands stored in one or more storage devices to achieve its functions. Furthermore, the system may be configured so that information sent from a terminal is processed by one or more computers located on the cloud, and the processing results are sent to the terminal.

The various calculations of each device (such as the gripping estimation device 17 and/or the controller 171) in the above-described embodiments may be executed in parallel using one or more processors, or using multiple computers via a network. Furthermore, the various calculations may be distributed to multiple processing cores within a processor and executed in parallel. Furthermore, some or all of the processes, means, etc. disclosed herein may be executed by at least one processor and storage device provided on a cloud that can communicate with the computer 81 via a network. In this way, each device in the above-described embodiments may take the form of parallel computing using one or more computers.

The processor 811 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.) including a computer control device and arithmetic device. The processor 811 may also be a semiconductor device including a dedicated processing circuit. The processor 811 is not limited to electronic circuits using electronic logic elements, but may also be realized by optical circuits using optical logic elements. The processor 811 may also include arithmetic functions based on quantum computing.

The processor 811 performs arithmetic processing based on data and software (programs) input from each device, etc., within the computer 81, and can output the arithmetic results and control signals to each device, etc. The processor 811 may control each component that makes up the computer 81 by executing the computer's OS (Operating System), applications, etc.

Each device in the above-described embodiments (such as the grip estimation device 17 and/or the controller 171) may be realized by one or more processors 811. Here, the processor 811 may refer to one or more electronic circuits arranged on a single chip, or to one or more electronic circuits arranged on two or more chips or two or more devices. When multiple electronic circuits are used, the electronic circuits may communicate with each other via wires or wirelessly.

The main memory device 812 is a memory device that stores instructions executed by the processor 811 and various data, and information stored in the main memory device 812 is read by the processor 811. The auxiliary memory device 813 is a memory device other than the main memory device 812. Note that these memory devices refer to any electronic component capable of storing electronic information, and may be semiconductor memory. The semiconductor memory may be either volatile memory or non-volatile memory. The memory device for saving various data in each device (such as the grip estimation device 17 and/or the controller 171) in the above-described embodiments may be realized by the main memory device 812 or the auxiliary memory device 813, or may be realized by internal memory built into the processor 811. For example, the memory unit 105 in the above-described embodiment may be realized by the main memory device 812 or the auxiliary memory device 813.

Multiple processors may be connected (coupled) to one storage device (memory), or a single processor may be connected. Multiple storage devices (memories) may be connected (coupled) to one processor. When each device (such as the grip estimation device 17 and/or the controller 171) in the above-described embodiments is composed of at least one storage device (memory) and multiple processors connected (coupled) to this at least one storage device (memory), it may include a configuration in which at least one of the multiple processors is connected (coupled) to at least one storage device (memory). This configuration may also be realized by storage devices (memories) and processors included in multiple computers. Furthermore, it may include a configuration in which the storage device (memory) is integrated with the processor (for example, a cache memory including an L1 cache and an L2 cache).

The network interface 814 is an interface for connecting to the communication network 82 wirelessly or via a wired connection. The network interface 814 may be an appropriate interface, such as one that conforms to existing communication standards. The network interface 814 may exchange information with an external device 83a connected via the communication network 82. The communication network 82 may be a WAN (Wide Area Network), LAN (Local Area Network), PAN (Personal Area Network), or any combination thereof, as long as information is exchanged between the computer 81 and the external device 83a. An example of a WAN is the Internet, an example of a LAN is IEEE 802.11 or Ethernet (registered trademark), and an example of a PAN is Bluetooth (registered trademark) or NFC (Near Field Communication), etc.

The device interface 815 is an interface such as USB that directly connects to the external device 83b.

External device 83a is a device connected to computer 81 via a network. External device 83b is a device connected directly to computer 81.

External device 83a or external device 83b may be, for example, an input device. The input device may be a device such as a camera, microphone, motion capture device, various sensors, keyboard, mouse, or touch panel, and provides acquired information to computer 81. It may also be a device equipped with an input unit, memory, and processor, such as a personal computer, tablet terminal, or smartphone.

Furthermore, external device 83a or external device 83b may be, for example, an output device. The output device may be, for example, a display device such as an LCD (Liquid Crystal Display), CRT (Cathode Ray Tube), PDP (Plasma Display Panel), or organic EL (Electro Luminescence) panel, or may be a speaker that outputs audio, etc. It may also be a device equipped with an output unit, memory, and processor, such as a personal computer, tablet terminal, or smartphone.

Furthermore, external device 83a or external device 83b may be a storage device (memory). For example, external device 83a may be network storage, and external device 83b may be storage such as an HDD.

Furthermore, external device 83a or external device 83b may be a device having some of the functions of the components of each device (such as gripping estimation device 17 and/or controller 171) in the above-mentioned embodiments. In other words, computer 81 may send or receive some or all of the processing results of external device 83a or external device 83b. For example, external device 83a or external device 83b may be at least one of the gripping device 11, observation device 13, weighing device 15, gripping estimation device 17, etc. in the above-mentioned embodiments.

When the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used in this specification (including the claims), it includes any of a, b, c, a-b, a-c, bc, or a-bc. It may also include multiple instances of any element, such as a-a, a-bb-b, a-a-bb-cc-c, etc. Furthermore, it also includes the addition of elements other than the listed elements (a, b, and c), such as a-bc-d, which includes d.

When expressions such as "data as input/based on/according to/in response to" (and similar expressions) are used in this specification (including the claims), unless otherwise specified, this includes cases where various data itself is used as input, or where various data that has been processed in some way (e.g., noise-added, normalized, intermediate representations of various data, etc.) is used as input. Furthermore, when it is stated that a result is obtained "based on/according to/in response to data," this includes cases where the result is obtained based solely on the data in question, as well as cases where the result is obtained as a result of being influenced by other data, factors, conditions, and/or states. Furthermore, when it is stated that "data is output," unless otherwise specified, this includes cases where various data itself is used as output, or where various data that has been processed in some way (e.g., noise-added, normalized, intermediate representations of various data, etc.) is output.

When the terms "connected" and "coupled" are used in this specification (including the claims), they are intended as open-ended terms that encompass direct connection/coupling, indirect connection/coupling, electrically connection/coupling, communicatively connection/coupling, functionally connection/coupling, and physically connection/coupling. These terms should be interpreted appropriately according to the context in which they are used, but any form of connection/coupling that is not intentionally or naturally excluded should be interpreted as being included within these terms without any restrictions.

When the expression "A configured to B" is used in this specification (including the claims), it may include the physical structure of element A having a configuration capable of performing operation B, and the permanent or temporary setting/configuration of element A being configured/set to actually perform operation B. For example, if element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and is configured to actually perform operation B through the setting of a permanent or temporary program (instruction). Furthermore, if element A is a dedicated processor or dedicated arithmetic circuit, etc., it is sufficient that the circuit structure of the processor is implemented to actually execute operation B, regardless of whether control instructions and data are actually attached.

When words implying containing or possessing (e.g., "comprising," "including," and "having") are used in this specification (including the claims), they are intended to be open-ended terms, including cases where something other than the object indicated by the object of the term is contained or possessed. When the object of such words implies no quantity or a singular number (e.g., expressions using the articles "a" or "an"), the expression should be construed as not being limited to a specific number.

In this specification (including the claims), although expressions such as "one or more" or "at least one" are used in some places and expressions that do not specify a quantity or that imply a singular number (expressions using the articles "a" or "an") are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or that imply a singular number (expressions using the articles "a" or "an") should be interpreted as not necessarily being limited to a specific number.

When this specification states that a particular advantage/result is obtained with respect to a particular configuration of a certain embodiment, it should be understood that the same advantage/result can also be obtained with one or more other embodiments having that configuration, unless otherwise stated. However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and/or states, etc., and that the effect is not necessarily obtained with that configuration. The effect is merely obtained with the configuration described in the embodiment when various factors, conditions, and/or states, etc. are satisfied, and the effect does not necessarily occur in claimed inventions that define that configuration or a similar configuration.

In this specification (including the claims), when multiple pieces of hardware perform a predetermined process, the pieces of hardware may work together to perform the predetermined process, or some of the hardware may perform all of the predetermined process. Furthermore, some of the hardware may perform part of the predetermined process, and other hardware may perform the rest of the predetermined process. In this specification (including the claims), when expressions such as "one or more pieces of hardware perform a first process, and the one or more pieces of hardware perform a second process" are used, the hardware performing the first process and the hardware performing the second process may be the same or different. In other words, it is sufficient that the hardware performing the first process and the hardware performing the second process are included in the one or more pieces of hardware. Note that hardware may include electronic circuits, devices including electronic circuits, etc.

In this specification (including the claims), when multiple storage devices (memories) store data, each of the multiple storage devices (memories) may store only a portion of the data, or may store the entire data.

Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, modifications, substitutions, and partial deletions are possible within the scope of the conceptual idea and spirit of the present invention, which is derived from the content defined in the claims and their equivalents. For example, in all of the above-described embodiments, when numerical values or formulas are used in explanations, they are shown as examples only and are not intended to be limiting. Furthermore, the order of each operation in the embodiments is shown as an example only and is not intended to be limiting.

Claims (7)

a parallel gripper for gripping food, the first and second members being opposed to each other and supported by a base portion so as to be openable and closable in a predetermined direction in a horizontal plane;
a gripping device including the parallel grippers;
a control means for controlling the operation of the gripping device so that the parallel grippers are inserted into the food from above in the vertical direction to grip a part of the food, and the gripped food is released at a target position;
The parallel gripper
the first member and the second member have a surface shape that forms a space in which the food is contained when the parallel grippers are in a closed state,
the first member and the second member face each other in the closed state and have a tip portion located below the base portion in the vertical direction,
The tip is formed with a straight edge,
a first surface shape of the first member intersecting with the predetermined direction is inclined from the base portion toward the tip portion toward the second member with respect to the vertical direction;
a second surface shape of the second member intersecting with the predetermined direction is inclined from the base portion toward the tip portion toward the first member with respect to the vertical direction;
the first surface shape and the second surface shape move in the predetermined direction in response to opening and closing of the parallel grippers.
The parallel gripper
system. the control means determines an area in which the food product is to be gripped within the area in which the food product is spread, and controls the operation of the gripping device so as to insert the parallel grippers into the determined area and perform the gripping.
The system of claim 1 . The control means determines to grab the food from an area where the state of the food satisfies a predetermined condition among the areas where the food is spread.
3. The system according to claim 1 or 2 . The control means acquires the state of the food in a plurality of areas among the areas in which the food is spread, and determines to grab the food from an area where the state of the food satisfies the predetermined condition.
The system of claim 3 . The state of the food product includes information related to the surface irregularities of the area where the food product is spread.
5. The system according to claim 3 or 4 . a measuring device for acquiring the state of the food in the area where the food spreads;
6. A system according to any one of claims 3 to 5 . the predetermined conditions include a condition regarding uncertainty of the amount of the food product grasped by the parallel grippers;
7. A system according to any one of claims 3 to 6 .