JP7504566B2 - Information processing method, robot system, article manufacturing method, program, recording medium, and information processing device - Google Patents

Description

The present invention relates to a control method for a robot system that picks up objects in a bulk pile, a method for manufacturing an article, a production system, a control program, a recording medium, and a robot system.

Conventionally, there have been known devices that pick up parts from a so-called bulk pile, where the parts are not positioned and are randomly placed on a tray or flat plate (see, for example, Patent Document 1 below). As shown in FIG. 17, for example, this type of pick-up device is composed of an image input device that captures images of a wide range, and a robot manipulator that picks up the parts and transports them to a specified placement position.

To ensure successful part removal in this type of removal device, it is necessary to determine in advance, for example, which part of the part the removal hand at the end of the robot manipulator should hold. This is because, if the hand does not hold the part in an appropriate position according to the part's shape when holding it, the part may fall off the hand or interfere with other parts. It is possible to determine the optimal part holding position in advance by performing a simulation in a virtual space.

However, in an actual robot manipulator, when the picking hand holds a part, an external force is applied to the part to hold it. This can cause the posture of the part to be picked to become distorted, and the robot manipulator may fail to hold it. In regard to this, as shown in, for example, Patent Document 2 below, it is conceivable to equip the robot manipulator with a function for learning the operating conditions when it fails to pick up a part during the period in which it actually operates the part.

Japanese Patent Application Laid-Open No. 5-127722 JP 2017-64910 A

However, in the above-mentioned Patent Document 2, a comprehensive search is performed for a part holding position that can hold the target part without changing its posture, which results in repeated failures and retries. This can result in a longer series of processing times (= cycle time) required for part manipulation, for example, picking up one part.

This is a problem specific to machine learning methods that actually operate a robot manipulator, because when a removal attempt fails, the only thing that is learned is to avoid the failed holding position for similar bulk pile patterns from the next time onwards. The control in Patent Document 2 is merely learning to randomly avoid failed positions. For example, the control in Patent Document 2 does not provide information such as which holding position should be avoided to hold the part without losing its posture regardless of external forces.

The objective of the present invention is to increase the success rate of parts removal in a short time after actual operation in production equipment such as a parts removal device, reduce the frequency of retry operations, and shorten the extension of cycle time.

One aspect of the present invention is an information processing method for acquiring a trained model to be used when an object is held by a robot , the information processing method comprising the steps of: acquiring information regarding the amount of movement of the object caused by contacting the object with the robot in order to hold the object, by simulation in a virtual environment; and acquiring the trained model by machine learning based on information regarding a plurality of the amounts of movement acquired corresponding to contact at different positions on the object .
Another aspect of the present invention is an information processing device that acquires a trained model to be used when an object is held by a robot , wherein a processing unit acquires information regarding the amount of movement of the object caused by contacting the object with the robot in order to hold the object by simulating in a virtual environment, and acquires the trained model by performing machine learning based on information regarding a plurality of the amounts of movement acquired corresponding to contact at different positions on the object .

According to the above configuration, instead of performing machine learning only on actual operations, a trained model is generated using a virtual environment to be used to determine candidate holding positions for actual picking operations. In addition, this trained model is retrained depending on whether or not the actual part picking operation is successful. Therefore, according to the above configuration, in production equipment such as a part picking device, it is possible to increase the picking success rate in a short time after actual operation, reduce the frequency of retry operations, and shorten the extension of cycle time.

1 is a block diagram showing a functional configuration of a component picking device according to the present invention; FIG. 11 is a flowchart showing a calculation process of the component picking device according to the present invention. (A) is an explanatory diagram showing an example of bulk parts, (B) is an explanatory diagram showing an example of the arrangement of bulk parts, (C) is an explanatory diagram showing an example of the part outer shape and division method, (D) is an explanatory diagram showing the arrangement of bulk parts when N = 3, and (E) is an explanatory diagram showing the setting of candidate holding positions. (A) is an explanatory diagram of an example of a bulk pile state in a virtual space, (B) is an explanatory diagram showing an example of point cloud data in a virtual space, (C) is an explanatory diagram showing an example of cropping of image data, and (D) is an explanatory diagram showing an example of the application of a virtual external force. 13A, 13B, and 13C are explanatory diagrams showing the manner in which an external force is applied in a suction type, an outer diameter gripping type, and an inner diameter gripping type. 1A is an explanatory diagram showing a data list of storage position candidates before and after ascending sorting for one pattern of image data, and FIG. 1B is an explanatory diagram showing a data list of storage position candidates before and after ascending sorting for all patterns of image data. FIG. 1 is an explanatory diagram showing an example of a learning model of a CNN (Convolutional Neural Network). FIG. 4A is a flowchart showing a process for acquiring image data, and FIG. 4B is an explanatory diagram showing a method for determining a part to be removed. FIG. 1 is an explanatory diagram showing an example of a CNN (Convolutional Neural Network) prediction model. 13 is an explanatory diagram showing an example of a process for changing the priority order of storage position candidates for one pattern of image data; FIG. FIG. 2 is a block diagram showing a functional configuration of a different component picking device according to the present invention; FIG. 12 is a flowchart showing a calculation process in the configuration of FIG. 11 . 12A is an explanatory diagram showing a method of calculating a judgment value in the configuration of FIG. 11, and FIG. 12B is an explanatory diagram showing a data list of image data and holding position candidates having judgment values in the configuration of FIG. 12A is an explanatory diagram showing an example of a learning model of a CNN (convoluted neural network) in the configuration of FIG. 11 , and FIG. 12B is an explanatory diagram showing an example of a prediction model of a CNN (convoluted neural network) in the configuration of FIG. 11 . 12A is a flow chart showing the process of acquiring image data in the configuration of FIG. 11, and FIG. 12B is an explanatory diagram showing an example of trimming of image data in the configuration of FIG. FIG. 12 is an explanatory diagram showing an example of a method for updating a judgment value in the configuration of FIG. 11 . 1 is an explanatory diagram showing the configuration of a production line for manufacturing articles using a part picking device according to the present invention; FIG. 12 is a block diagram showing an example of a control system constituting a calculation unit, a learning unit, etc. in FIG. 1 or FIG. 11.

Below, a description will be given of an embodiment of the present invention with reference to the attached drawings. Note that the configuration shown below is merely an example, and those skilled in the art can appropriately change the detailed configuration, for example, without departing from the spirit of the present invention. Also, the numerical values used in this embodiment are for reference only and do not limit the present invention.

Figure 1 is a schematic diagram showing an embodiment of a robot system according to this embodiment, in particular a robot system configured as a part picking device 1. The part picking device 1 is composed of a calculation unit 100, a learning unit 200, a robot manipulator 11 that picks parts in an actual device, and a picking result detection unit 15 that detects the part picking results of the robot manipulator 11. The calculation unit 100 has a function of determining the prioritization of candidate holding positions for virtual parts piled up in a virtual space. The learning unit 200 has a function of determining the prioritization of part holding positions when the robot manipulator 11 picks parts from the actual parts piled up, based on the calculation results of multiple trials performed by the calculation unit 100.

The robot manipulator 11 of the robot system in FIG. 1 can be arranged in a production line (production system) for producing items, for example, in a form as shown in FIG. 17. The robot manipulator 11 in FIG. 17 is delivered by another transport means (not shown) such as a conveyor. Then, the parts 13 randomly piled on a flat plate (e.g., a parts supply tray) 14 are picked up one by one, placed on a jig 150, and handed over to a manufacturing process carried out by other production equipment.

Figure 18 shows an example of a specific hardware configuration of a control system that constitutes the calculation unit 100, learning unit 200, etc. in Figure 1. The control system shown in the figure can be realized, for example, in a so-called PC form implementation.

The control system of FIG. 18 can be configured with a CPU 1601 as a main control means, a ROM 1602 as a storage device, and PC hardware equipped with a RAM 1603. The ROM 1602 can store the control program and constant information of the CPU 1601 for implementing the control procedure of this embodiment. The RAM 1603 is used as a work area for the CPU 1601 when executing the control procedure. An external storage device 1606 is also connected to the control system of FIG. 18. The external storage device 1606 is not necessarily required for implementing the present invention, but can be configured with an HDD, SSD, or an external storage device of another system mounted on a network.

The control program of the CPU 1601 for implementing the arithmetic processing of this embodiment, for example, the processing of the calculation unit 100 and the learning unit 200, is stored in a storage unit such as the external storage device 1606 or the ROM 1602 (for example, an EEPROM area). In this case, the control program of the CPU 1601 for implementing the control procedure of this embodiment can be supplied to each of the above storage units via the network interface 1607 (NIF) and updated to a new (different) program. Alternatively, the control program of the CPU 1601 for implementing the control procedure described below can be supplied to each of the above storage units via various storage means such as magnetic disks, optical disks, and flash memories and their drive devices, and the contents can be updated. Various storage means, storage units, or storage devices in a state in which the control program of the CPU 1601 for implementing the control procedure of this embodiment is stored constitute a computer-readable recording medium storing the control procedure of the present invention.

The network interface 1607 can be configured using a communication standard for wired communication such as IEEE 802.3, or wireless communication such as IEEE 802.11 and 802.15. The CPU 1601 can communicate data with other devices 1104 via the network interface 1607. These other devices 1104 correspond to, for example, controllers or overall control devices related to the manufacture of goods, or other management servers. Communication with the robot manipulator 11 in FIG. 1 and other production equipment arranged on the production line can also be performed via the network interface 1607.

The control device in FIG. 18 also includes a user interface device 400 (UI device). The user interface device 400 may be a GUI device consisting of an LCD display, a keyboard, a pointing device (mouse, joystick, etc.), etc. Through this user interface device 400, the progress of various arithmetic processes can be notified to the user, or various parameters that control the processes described below can be set. The user interface device 400 can also be used as a means for the user to specify holding position candidates described below, for various user notifications, or for inputting user commands related to the progress of processes.

(Calculation section)
2 shows a process flow for determining a specific bin pile form and the priority of the corresponding part holding positions in a virtual space (virtual environment) executed by the calculation unit 100. In the process in Fig. 2, first, in step S101, the virtual bin pile state generation unit 101 generates a bin pile state of parts (virtual objects) in the virtual space (virtual bin pile state generation process).

Naturally, the parts generated as virtual objects in this virtual space correspond to parts of the objects handled by the robot manipulator 11 on the actual production line. For example, parts as virtual objects generated in the virtual space and real parts have the same shape, size, and weight. These virtual parts can be generated from design data of the parts handled by the robot manipulator 11 in the actual production process, for example, using CAD (Computer Aided Design). The number N of parts generated for the virtual bulk state can be freely set. For ease of explanation, the following will mainly be explained using an example where N=2, i.e., a bulk state of two parts.

In generating the bulk pile state in step S101, as shown in FIG. 3(A), parts with the same shape, weight, and material as the parts to be actually removed are generated in virtual space, and designated as fixed part 130 and movable part 131, respectively. Movable part 131 is placed on top of fixed part 130.

Figure 3 (B) shows the relative positions of the fixed part 130 and the moving part 131. The center of gravity G0 of the fixed part 130 is located at the origin O of the virtual space, and the center of gravity G1 of the moving part 131 is located at a position shifted by Δx and Δy from the center of gravity G0 of the fixed part 130. In this case, the constraint conditions for Δx and Δy are shown in the following formula (1).

Δｘ、Δｙについては図３（Ｃ）に示すように、部品の外形に対しｎ分割、ｍ分割した幅に基づき、下式（２）、（３）のように決定する。

As shown in FIG. 3C, Δx and Δy are determined based on the width of the outer shape of the part divided into n and m, as shown in the following formulas (2) and (3).

Based on the above expression, if the center of gravity position G1 of the moving part 131 is arranged as (Δx, Δy), (-Δx, Δy), (Δx, -Δy), (-Δx, -Δy), a total of n×m random pile arrangement patterns can be obtained. Furthermore, in each arrangement, if the moving part 131 is rotated by an amount Δθ obtained by dividing 360 degrees by k on the x-y plane shown in Figure 3(B) as shown in the following formula (4), a total of n×m×k random pile arrangement patterns can be obtained. For example, when n=10, m=10, k=12, a total of 1200 random pile arrangement patterns can be obtained.

By the above-mentioned calculation process, the virtual random pile state generating unit 101 can generate random pile states in the same manner even when the number of parts is N=3 or more. For example, when N=3, the random pile arrangement state (pattern) is formed by arranging a movable part 132 on top of a movable part 131 as shown in FIG. 3(D). In this case, if the movable part 131 is regarded as a fixed part and the movable part 132 is moved in the same manner as described above, n×m×k can be obtained, and when combined with the arrangement pattern of the fixed part 130 and the movable part 131, a total of (n×m×k) ² patterns of arrangement can be obtained. If the number of parts is N, the calculation is repeated recursively, so the number of arrangement patterns is the (N-1)th power of the number of parts and patterns, as shown in the following formula (5).

Next, in step S102 of FIG. 2, the component holding position candidate generation unit 102 outputs holding position candidates when the hand 12 of the robot manipulator 11 (virtual robot) holds the component. As shown in FIG. 3(E), holding position candidates H1 to H4 are provided at multiple points on the surface of the component so that the contact surface between the hand 12 and the component is equal to or larger than a certain area. For example, when a suction pad (suction head) is used for the hand 12, the surface where the suction pad and the surface of the component can be in close contact with each other without any gaps is set as the holding position candidate. When holding with a suction pad (suction head), uneven surfaces of the component and positions of holes are not preferable as holding position candidates because gaps are created between the suction pad and the component.

These holding position candidates are determined, for example, by a designer when designing the hand 12, and the part holding position candidate generation unit 102 sequentially outputs the holding position candidates based on that. For example, based on the design information of a part that is an actual object, the part holding position candidate generation unit 102 arranges multiple holding position candidates for that object around its center of gravity. The detailed aspects of generating these holding position candidates are arbitrary, as described below.

In addition, in the part holding position candidate generation process executed by the part holding position candidate generation unit 102, multiple holding position candidates for the object to be prioritized in the learned model can be generated based on the design information of the actual part. Alternatively, multiple holding position candidates for the object may be generated based on the holding position specified by the user via the user interface device 400. In this case, the availability of the holding position specified by the user via the user interface may be determined based on the design information of the actual object, and if the holding position specified by the user is unavailable, the user may be notified of this. For example, in the case of using a suction pad, positions where air leakage is likely to occur, such as the positions of holes and steps, can be compared with the design information of the actual object. Then, it can be checked whether the user has specified such an inappropriate holding position.

Next, in step S103 of FIG. 2, one storage location is selected from the storage location candidates generated in step S102. This selection process is repeated until all storage location candidates have been selected in step S107, which will be described later.

Next, in step S104, the virtual part state recognition unit 103 recognizes the parts in the random pile state generated in step S101 in the virtual space. As shown in FIG. 4(A), by plotting the height information of the part surface on the x-y plane along the Z axis shown in the coordinate system in the virtual space, point cloud data as shown in FIG. 4(B) can be obtained. Such point cloud data is recorded in a storage device such as the RAM 1603 (FIG. 18) using an image data format such as TIFF (Tagged Image File Format) format that can hold "pixel data + height information".

Furthermore, for consistency with the image data of the recognition unit 10 described below, this image data is trimmed and recorded so that the center of gravity of the topmost moving part (moving part 132 in the example of FIG. 3(D)) is the center and the outline of the part is included. When trimming, as shown by Wx and Wy in FIG. 3(C), fixed values are set, such as the number of pixels that will fit on the screen at twice the size of the part outline, and the aspect ratio is also set to an arbitrary constant.

Next, in step S105, the virtual part pick-up operation execution unit 104 in FIG. 1 causes a virtual robot simulating the actual robot manipulator 11 in the virtual environment to perform a part-holding operation (virtual part-holding operation execution process). The virtual robot simulating the robot manipulator 11 can be modeled using design information of the robot manipulator 11, etc. In the pick-up operation in the virtual environment by this virtual robot, as shown in FIG. 4(D), in the virtual part-holding operation, an external force that is assumed to be applied to the part when the robot manipulator 11 actually holds the part is virtually applied to one of multiple holding candidates. In this procedure, it is necessary to consider the frictional force and normal force between the parts, and the frictional force and normal force between the part and the installation surface, and the calculation can be performed using a computer-aided tool such as CAE (Computer Aided Engineering).

As shown in FIG. 5A, the way in which the hand attached to the tip of the robot manipulator applies force when holding a part is set appropriately according to the shape of the hand 12 attached to the tip of the robot manipulator 11. For example, if an adhesive suction pad is used for the hand 12, the direction and magnitude of the force of the hand 12 can be set to a maximum of 5 N from the normal direction of the contact surface where the hand 12 and the part come into contact to the direction in which the part is pressed. The area to which the hand 12 applies force can be set to be equal to the area of the suction pad, and the pressing stroke amount can be set to 1 mm. Similarly, in the case of hands using an external gripping method (121) or an internal gripping method (122) other than the adhesive method as shown in FIG. 5B and FIG. 5C, the magnitude and direction of the force, the area, and the pressing stroke amount are each set.

Next, in step S106 of FIG. 2, similar to step S104, the virtual part state recognition unit 103 recognizes the position of the part after the virtual external force is applied in step S105, and the virtual part movement amount calculation unit 105 calculates the movement amount of the part before and after the removal operation. The virtual part movement amount calculation unit 105 obtains the center of gravity position of each part from the position information of the part in the x-y-z coordinate system in the virtual space recognized in steps S104 and S106, and calculates and records the displacement Δ of the center of gravity position in the x-, y-, and z-directions, for example, as shown in the following formula (6).

2, it is confirmed whether the trials of steps S103 to S106 have been completed for all the holding position candidates generated in step S102. If the trials of steps S103 to S106 have not been completed for all the generated holding position candidates, the process returns from step S107 to step S103, an untried holding position candidate is selected, and the process of steps S103 to S106 is repeated. If the trials of steps S103 to S106 have been completed for all the generated holding position candidates, the process proceeds from step S107 to step S108.

In step S108, the priority determination unit 106 determines the priority of all holding position candidates based on the Δ calculated by equation (6). When determining the priority, as shown in FIG. 6(A), for example, for pattern 1 of the bulk pile state determined in step S101, the holding position candidates are sorted in ascending order by Δ, and the smaller Δ is defined as the higher the priority. At this time, if there are holding position candidates with the same Δ, the holding position candidate with the smaller number is determined to have the higher priority. This priority determination is based on the inference that the smaller the movement amount (Δ) calculated when a part is virtually held, the more stable the holding and removal can be performed.

In step S109 of FIG. 2, the sorting calculation result of the priority order calculated in step S108 is recorded in a storage device such as the RAM 1603 (FIG. 18). Next, in step S110, it is confirmed whether the trials of steps S101 to S109 have been completed for all arrangement patterns generated in step S101. If not completed, the process returns to step S101, and an untried random pile pattern is generated, and the process of steps S102 to S109 is executed. If completed, the process flow of the calculation unit 100 ends. At the end of this process flow, as shown in FIG. 6B, a list in which the priority order of the holding position candidates is determined for all random pile patterns is output. This list is saved in a storage device (not shown) of the learning unit 200.

(Learning Department)
1, the configuration of a learning unit 200 that generates a trained model used to derive a priority list of holding position candidates for a target part from an image recording the position and orientation of the target part will be described. The learning unit 200 is composed of a priority learning unit 201, a priority prediction unit 202, and a priority update unit 203. First, the priority learning unit 201 statistically analyzes the relationship between a huge amount of input data and output data based on the combination. Then, it has a function of modeling certain rules to obtain a trained model (priority learning process), which is so-called machine learning.

Machine learning includes "supervised learning," in which a learned model is calculated from training data in which the relationship between input and output data is known, and "unsupervised learning," in which certain rules common to input data are modeled when the output has not been determined. "Reinforcement learning," in which the next output content is determined from the current input data, is also known. The machine learning used in this embodiment is "supervised learning," in which a model is calculated from a data set (learning data) in which the input and output data are known.

In the "supervised learning" of the priority learning unit 201, the image data obtained by the virtual part state recognition unit 103 in step S104 is used as input data as the supervised data, and the priority of the holding position candidates obtained in step S108 is used as output data (FIG. 6B). When performing "supervised learning" using such image data as input, for example, a neural network such as CNN (convoluted neural network), which is known as one of deep learning, can be used. This CNN (convoluted neural network) configures a trained model that associates a specific position and posture of a virtual object with the priority of multiple holding positions that the virtual robot device can take. By using such a trained model, it is possible to output the priority of multiple holding positions that the robot device can take for a specific position and posture of the object from the specific position and posture of the object.

In this embodiment, for example, a convolutional neural network (CNN) as shown in FIG. 7 is used, but other forms of CNN may also be used. The CNN in FIG. 7 constitutes a network that can arrive at a list of priorities of storage position candidates in the output section from image data in the input section as a trained model. The network is composed of multiple stages of convolution layers, pooling layers, and dropout layers that process image data in the input section. By using such a priority learning unit 201, it is possible to model the relationship between image data and the priorities of storage position candidates.

2 and the machine learning of the trained model using the virtual environment by the priority learning unit 201 (learning unit 200) (priority learning process) is completed, the user is notified of the completion via the user interface device 400. This notification can be performed by displaying text or audio. After the trained model is generated using the virtual environment, a trial of the removal operation by actual robot control can be performed. Therefore, the user can be notified that the removal operation can be performed by the robot control unit via the user interface device 400. In this case, the user interface device 400 can also be configured to input a user operation that commands the removal operation by the robot control unit. The user interface device 400 can be configured so that the user operation that commands the removal operation can be performed by keyboard operation, pointing device operation, audio input, etc.

After generating the trained model using the virtual environment, a trial of the picking operation by actual robot control can be performed (robot control process). In this actual picking operation, the priority prediction unit 202 uses the trained model calculated by the priority learning unit 201 to predict the priority of the holding position when the robot manipulator 11 picks up a part in the actual device. In predicting this priority, image data of the parts in a random pile in the actual device, acquired by the recognition unit 10, is used as input data. The image data used here is, for example, point cloud data containing three-dimensional information equivalent to that obtained by the virtual part state recognition unit 103.

For this reason, the recognition unit 10 (Figure 1) needs to be configured to obtain three-dimensional information that reflects the position and orientation of the part. For example, the recognition unit 10 can be configured using a stereo camera as a three-dimensional measurement means that can obtain point cloud data that includes the above-mentioned three-dimensional information. Alternatively, instead of a stereo camera, the recognition unit 10 may use a three-dimensional measuring device that combines a single camera and a projector.

Figure 8 (A) shows the flow of the image data acquisition process by the recognition unit 10, which is necessary for actual robot control. The process in Figure 8 (A) is executed when actually controlling the robot to pick up a part using the trained model that has been trained as described above, as shown in Figure 7.

In step S201 of FIG. 8(A), imaging processing is performed using, for example, a stereo camera constituting the recognition unit 10, and in step S202, 3D measurement processing is performed to generate 3D point cloud data in a format equivalent to that of FIG. 4(B). In step S203, the generated 3D point cloud data is used to recognize the imaged part. This part recognition can use a method such as template matching using a 3D CAD model or gray-value-based normalized cross-correlation matching. This allows the position and orientation of the part to be removed to be recognized.

In step S204, when multiple parts are recognized in step S203, the part to be removed is determined. As shown in FIG. 8B, when two target parts 134 and 135 (overlapping) are recognized as the parts to be removed, the overlapping and interference between the parts are checked, and the part that is easiest for the robot manipulator 11 to remove is determined. For example, in the case of FIG. 8B, in order to preferentially remove the part with the smallest overlapping area between the parts, target part 134 is selected because the overlapping area between the parts is smaller than that of target part 135. As another method of selecting the target part, height information included in the three-dimensional information may be used to give priority to the part at the top of randomly arranged parts in a pile, or control may be performed to select parts in order of highest recognition accuracy by matching.

Next, in step S205, the 3D point cloud data is trimmed around the center of gravity of the part to be extracted, and in step S206, it is converted to a specified image file format, for example TIFF format, and recorded. When trimming, it is preferable to use the same number of pixels and aspect ratio as when trimming was performed by the virtual part state recognition unit 103. However, if these same settings cannot be applied, it is desirable to adjust the trimmed image file to have the same number of pixels and aspect ratio by performing pooling, which resamples the resolution, or by performing interpolation or interpolation processing between pixels.

In this manner, the image data of the trimmed part is input to the priority prediction unit 202, and the learned model (CNN in FIG. 9) calculated by the priority learning unit 201 is used to predict the priority of the part, i.e., which position the part should be held at. The initial state of the network in FIG. 9 is the same as the learned model calculated by the priority learning unit 201 as in FIG. 7. Using this learned model (CNN in FIG. 9), it is possible to arrive at a priority list of candidate holding positions for the part in the output unit from an image of the part actually captured in the input unit. Then, the robot control unit controls the robot manipulator 11 to hold the part holding position with the highest priority obtained using the learned model (CNN in FIG. 9), and the robot manipulator 11 executes the removal operation.

After executing the removal operation, the robot manipulator 11 performs the part holding operation in accordance with the instructions of the priority prediction unit 202, and transports the part to a specified position while still holding the part. At the destination, the removal result detection unit 15 checks the removal result. The removal result detection unit 15 checks, for example, whether the robot manipulator 11 is holding a part at the destination, and outputs a binary signal indicating removal success if the part is being held, or removal failure if the part is not being held.

The part that detects the presence or absence of these parts can be, for example, a photoelectric sensor or a laser-type object sensor, or in particular, in the case where the hand 12 is of a suction type, the pressure value at the time of suction, etc. In this embodiment, if the removal result detection part detects a removal failure, the priority update part 203 performs a re-learning process to update the model created by the priority learning part 201 according to the result of this failure (priority update process).

For example, if the priority prediction unit 202 has output the priority prediction value (upon failure) shown in FIG. 10, the storage position candidate used for the failed retrieval, i.e., the top candidate in the priority list, is dropped to the end (191->192). Then, a list after priority change is generated in which the other priorities are raised one by one. This list is added to the list shown in FIG. 6 (B), and re-learning is performed by the priority learning unit 201, updating the learned model.

As described above, according to this embodiment, the trained model generated using the virtual environment is re-learned for similar part stacking patterns (positions and postures) from the next time onwards, depending on the results of the picking operation actually attempted using the trained model. Therefore, in order to avoid repeating the same failure, the trained model for reaching the priority order of the candidate holding positions can be updated depending on the results of the picking operation attempted in the real environment. Therefore, after the picking operation of the robot manipulator 11 is actually started, the picking success rate can be increased quickly, and the extension of the cycle time due to the retry operation can be shortened.

In addition, in part removal using the trained model of this embodiment, a virtual environment is utilized, making it possible to easily generate an original trained model quickly and easily without actually operating the removal hardware. Moreover, after actually operating the removal hardware, re-learning is performed to update the trained model that controls part removal according to the actual part removal results. Therefore, the trained model that controls part removal can be grown to be able to control part removal with high accuracy, reflecting various conditions in the real environment, and removal operations can be performed with a high success rate in the real environment.

That is, according to the above configuration, a trained model capable of determining the priority order of candidate holding positions for part picking in a virtual environment can be generated in advance. Therefore, the robot manipulator can be equipped with a function for predicting a holding position for a single part to be picked that will not lose its posture even when an external force is applied. Therefore, the success rate of part picking can be increased in a short time after the start of operation in the real environment. Furthermore, even if a part fails to be picked for a specific pattern of random pile conditions, the part holding position is changed to another candidate holding position with a relatively high success rate for similar patterns of random pile conditions from the next time onwards. By performing such re-learning, it is possible to prevent the same picking failure from being repeated, reduce the frequency of part picking retry operations, and shorten the extension of the cycle time, which is an excellent effect.

(Select parts and holding positions with high removal success rate)
It is also possible to control the selection of a component and a holding position with a high success rate of removal. In the above, an example was shown in which a holding position with the highest success rate of removal was selected for one component recognized by the recognition unit 10, but here, the component with the highest success rate of removal is selected from among a plurality of components. Furthermore, the holding position with the highest success rate of removal among the selected parts is selected and the component is removed. This control will be described with reference to Figs. 11 and 12. Fig. 11 shows the configuration of the system corresponding to Fig. 1 above, and Fig. 12 shows the control procedure corresponding to Fig. 2 above. Fig. 11 differs from Fig. 1 in that a holding success/failure determination unit 106b, a holding success rate learning unit 201b, a holding success rate prediction unit 202b, and a holding success rate update unit 203b are provided.

In the calculation unit 100 in FIG. 11, the same processing as in FIG. 2 is performed until the virtual part movement amount calculation unit 105 calculates the part movement amount in step S106. In the control in FIG. 12, after step S107 for determining whether all holding position candidates have been processed, the holding success/failure determination unit 106b determines whether holding is successful in step S108b. For example, as shown in FIG. 13(A), the holding success/failure determination unit 106b determines whether holding is successful based on Δ calculated in (Equation 6) and an arbitrary threshold value Δth, as shown in the following equation (7).

When the calculation process flow in FIG. 12 is completed, a list of calculation results like that in FIG. 13(B) is output and stored in the learning unit 200.

Next, the learning unit performs "supervised learning" on the retention success rate using the retention success rate learning unit 201b based on the list of calculation results described above. At this time, as the teacher data, the image data recognized in step S104 is used as the input data, and the pattern number (the leftmost column in FIG. 13(B)) is used as the output data. Through this learning process, the relationship between the two can be modeled into a CNN, a learned model, as shown in FIG. 14(A).

When actually attempting to pick up a part, the holding success rate prediction unit 202b predicts the part that the robot manipulator 11 will pick up in the actual device and the holding position where it should be held, based on the learned model calculated by the holding success/failure determination unit 106b described above, as shown in Figure 14 (A).

To make predictions, the input data is acquired using a recognition unit 10 implemented with a stereo camera or the like in the same manner as described above, and image data of the actual parts in a randomly stacked state is acquired, and this is used as input data for the trained model.

The process of acquiring this image data is shown in FIG. 15(A). FIG. 15(A) corresponds to the process of FIG. 8(A), and differs from FIG. 8(A) in that step S204 in FIG. 8(A) is replaced by step S204b. In the above example, the target part was narrowed down to one depending on the overlapping of the parts, etc. (FIG. 8(B)), but in step S204b in FIG. 15(A), for example, multiple target parts can be selected. When multiple target parts are determined, the 3D data for the number of target parts is trimmed and recorded (steps S205 and S206) as shown by 136 and 137 in FIG. 15(B), just as in the case where there is only one target part.

Next, as shown in FIG. 14(B), the holding success rate prediction unit 202b uses the learned model learned by the holding success rate learning unit 201b to infer which pattern the image is closest to for the input image data, and outputs the number of the closest pattern. Then, according to the output pattern number, the judgment values in the list shown in FIG. 13(B) are compared, and the robot manipulator 11 is controlled to hold the holding position candidate with the largest judgment value for the removal success rate (minimum value 0, maximum value 1). At this time, if multiple image data are input, the pattern numbers are inferred for all image data, the holding position with the largest judgment value of all judgment values is selected, and the robot manipulator 11 is caused to attempt to hold the part.

The robot manipulator 11 performs the part pick-up operation at the position instructed by the holding success rate prediction unit 202b, and similarly to the above example, the pick-up result detection unit 15 detects whether the part was picked successfully or not. If the part pick-up operation fails, the holding success rate update unit 203b performs re-learning to update the model created by the holding success rate learning unit 201b. For example, if the holding success rate prediction unit 202b predicts the pattern number shown in FIG. 16 from the input data, holding position candidate 1 is selected. The robot manipulator 11 performs the part pick-up operation according to this instruction, and if the pick-up result detection unit 15 detects a pick-up failure, re-learning is performed to update the judgment value as shown in the list in FIG. 16 (281->282). The judgment value can be updated, for example, by a calculation such as the following formula (8).

This newly updated pattern is added to the list in FIG. 16, and the model is re-learned in the holding success rate learning unit 201b. As described above, with the configurations in FIG. 11 to FIG. 16, it is possible to learn the success/failure of picking for similar random pile patterns from the next time onwards, taking into account the part picking success rate, so as not to repeat the same failure. Therefore, after the robot manipulator 11 actually starts the picking operation, it is possible to quickly increase the picking success rate, and shorten the extension of the cycle time due to the retry operation.

In other words, with the above configuration, the robot manipulator can select the part with the highest success rate of removal and the holding position from among the parts in a random pile, and then remove the part. Therefore, in addition to the above-mentioned effects, there is the excellent effect that the target part can be removed with a higher success rate.

1...part picking device, 10...recognition unit, 11...robot manipulator, 12...hand, 13, 130, 131...part, 14...plate, 15...picking result detection unit, 100...calculation unit, 101...virtual bulk pile state generation unit, 102...part holding position candidate generation unit, 103...virtual part state recognition unit, 104...virtual part picking operation execution unit, 105...virtual part movement amount calculation unit, 106...priority determination unit, 106b...holding success/failure determination unit, 200...learning unit, 201...priority learning unit, 201b...holding success rate learning unit, 202...priority prediction unit, 202b...holding success rate prediction unit, 203...priority update unit, 203b...holding success rate update unit.

Claims (27)

An information processing method for acquiring a trained model used when holding an object by a robot, comprising:
acquiring information on an amount of movement of the object caused by contact of the robot with the object in order to hold the object, by simulating the information in a virtual environment;
and acquiring the trained model by performing machine learning based on information about a plurality of the movement amounts acquired in response to contact at different positions of the object.
23. An information processing method comprising: 2. The information processing method according to claim 1,
a user sets the different position on a virtual object that virtually represents the object;
23. An information processing method comprising: 3. The information processing method according to claim 2,
If there is a position where the virtual robot virtually representing the robot cannot hold the virtual object among the different positions set by the user, the user is notified of this.
23. An information processing method comprising: 4. The information processing method according to claim 1,
The trained model outputs a priority order for the different positions when holding the object, and the robot is controlled based on the priority order.
23. An information processing method comprising: 5. The information processing method according to claim 4 ,
Using the robot to hold the object based on the trained model;
updating the priority based on a result of holding the object by the robot;
23. An information processing method comprising: 6. The information processing method according to claim 5 ,
When an attempt is made to hold the object at the position with the highest priority among the priorities, and the object cannot be held, the priority of the position with the highest priority is lowered, and the other priorities are raised one by one.
23. An information processing method comprising: 7. The information processing method according to claim 5 ,
acquiring a position and orientation of the object using an imaging device, and acquiring the different position candidates and the priority order in the position and orientation using the trained model;
23. An information processing method comprising: 8. The information processing method according to claim 7 ,
The imaging device is capable of three-dimensional measurement, and there are at least two objects;
Among the objects, an object having the smallest area of interference between the objects, an object having the highest height information, or an object having the highest recognition rate by the imaging device is selected as the object to be held preferentially.
23. An information processing method comprising: 9. The information processing method according to claim 1,
the different positions are set in at least two different positions and orientations of the object,
23. An information processing method comprising: 5. The information processing method according to claim 4,
The machine learning is performed so as to determine the priority order in ascending order of the movement amount.
23. An information processing method comprising: 2. The information processing method according to claim 1,
a virtual external force is applied to a virtual object virtually representing the object by a virtual robot virtually representing the robot at each of the different positions to obtain the amount of movement.
23. An information processing method comprising: 2. The information processing method according to claim 1,
A judgment value indicating whether or not a virtual object virtually showing the object at the different positions is held is obtained, and the trained model capable of outputting the object with a high holding success rate and its position is obtained by the machine learning based on the judgment value.
23. An information processing method comprising: 2. The information processing method according to claim 1,
setting the different positions based on design information of the object;
23. An information processing method comprising: The information processing method according to claim 13 ,
The different positions are set around the center of gravity of the object.
23. An information processing method comprising: The information processing method according to any one of claims 1 to 14 ,
The machine learning uses a convolutional neural network.
23. An information processing method comprising: 2. The information processing method according to claim 1,
acquiring an image of a state of a virtual object when a virtual robot representing the robot is brought into contact with a virtual object representing the object in the virtual environment, and using the image for the machine learning;
23. An information processing method comprising: The information processing method according to claim 16 ,
An image having the same pixel count and aspect ratio as the image is acquired by an imaging device that captures the position and orientation of the object.
23. An information processing method comprising: 18. The information processing method according to claim 1,
Notifying the user that acquisition of the trained model has been completed;
23. An information processing method comprising: 20. The information processing method according to claim 18 ,
notifying the user that acquisition of the trained model has been completed and that holding of the object by the robot can be executed;
23. An information processing method comprising: 20. The information processing method according to claim 1,
The object is bulk loaded.
23. An information processing method comprising: The information processing method according to any one of claims 1 to 20 ,
The amount of movement is information about a position or a posture of the object that has changed from before the robot contacts the object in the simulation.
23. An information processing method comprising: 2. The information processing method according to claim 1,
In the simulation, information is obtained regarding the amount of movement in each of a case where a virtual object virtually representing the object is held at a first position by a virtual robot virtually representing the robot, and a case where the virtual object is held at a second position different from the first position by the virtual robot.
23. An information processing method comprising: A robot system comprising the robot controlled using the trained model acquired by the information processing method according to claim 1 . A method for manufacturing an article, comprising the steps of: manufacturing an article using the robot system according to claim 2 or 3 . A program for causing a computer to execute the information processing method according to any one of claims 1 to 22 . A computer-readable recording medium storing the program according to claim 25 . An information processing device that acquires a trained model used when holding an object by a robot,
The processing unit:
acquiring information on an amount of movement of the object caused by contact of the robot with the object in order to hold the object, by simulating the information in a virtual environment;
and acquiring the trained model by performing machine learning based on information about a plurality of the movement amounts acquired in response to contact at different positions of the object.
23. An information processing apparatus comprising:

Images